Disclosure per rule 1: I work at Schema Labs and one of the three models tested is ours<\/p>\n
The thing that started this was annoyance rather than research. most datasets any of us actually work with show up with columns called V1 through V57, or metric_14, or some four character code from a system nobody maintains. meanwhile benchmarking mostly happens on clean data where the age column is called age, which quietly rewards a model for reading the header instead of the values.<\/p>\n
so we took 20 numerical classification datasets off OpenML and ran each twice. once as published, once with every column name stripped. same splits otherwise.<\/p>\n
mean ROC-AUC with names gone:<\/p>\n
Schema-2: 0.9230
TabuLa-8B: 0.8658
ConTextTab: 0.8541<\/p>\n
ours held flat, 0.9230 with names and 0.9230 without. the other two both gave up roughly 7 points once the headers went. flagging that these are our internal runs, not third party replicated.<\/p>\n
one thing that doesn’t fit the story but is true: stripping names doesn’t strip position, and column order still carries signal on some of these. we didn’t control for it and probably should have.<\/p>\n
all 20 are public on OpenML so anyone can rerun this against a model we didn’t cover. would like someone to get a different number. if you want the full per-dataset breakdown and the methodology, comment and I’ll send it over.<\/p>\n<\/div>\n
submitted by \/u\/No-Plant-5234 <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
