disclosure per rule 1: i work at Schema Labs. we train models on tables instead of text. there is a link at the bottom because this sub allows one. i am here for the answer to the question at the end.
here is the claim with the numbers attached because i would rather be argued with than believed.
take a standard tabular benchmark. strip every header off it. replace price and age and zip with positional tokens so there is nothing left but values. comparable models drop about 7 points of mean ROC-AUC. ours goes 0.9224 to 0.9230. flat.
that is not a claim that we win on clean data. those models start ahead of us when the headers are good. it is a claim that we do not read the headers at all, which only matters because production tables do not have good ones. they have val_b and metric_14 and a four-character code from a system nobody has logged into since 2019.
two more. on sector identification we name the correct industry of a dataset we have never seen, from raw values alone, at 86.3% top-1 out of 10,000 possible sectors. on multi-table entity matching we lead published state of the art across all six standard benchmarks with zero shared keys between tables, including a 1.9x improvement over the previous best on one of them. on the geospatial benchmark our lead is 0.07 F1 which i would not want read as more than it is.
every one of those is our own harness, run under each benchmark’s published protocol, one sealed configuration, no per-dataset tuning. competitor figures are third-party published values we did not re-run. nobody independent has replicated us. we are not on the public leaderboard because entering it requires shipping a runnable wrapper and we do not distribute weights. i put all of that up front every time rather than waiting for someone to ask.
which is the actual question.
we are closed. no pip install, no weights, no local option. none of that changes for at least six months. to try it you make an account, you put a card on file, you upload your data. i think that single fact is the biggest thing standing between us and everyone reading this.
so would you use a thing like this at all. not would you like it to exist, everybody likes things existing. would you upload a real table to a hosted model you cannot inspect, run by a company you had not heard of ninety seconds ago. if the answer is no i want to know what moves it. a named customer. a SOC 2. a benchmark you ran yourself on your own data. an on-prem story you would never actually take up but need to hear. or is the honest answer that nothing moves it and closed is closed.
and if you would rather test the claim than discuss it: send me two tables nobody at your company can explain. no account, no card, nothing to sign up for. i run them and send back what each column appears to be with a confidence score on every one. i flag the ones we got wrong as well, because those are the interesting half and you should see them before you decide anything.
anonymised is fine. structure and distributions are what matter.
submitted by /u/No-Plant-5234
[link] [comments]
