The core idea: take a real document (SEC filings, contracts, enterprise email), verify by exhaustive normalized scan that a specific plausible fact is not in it, then ask about that fact. The honest answer is \u201cnot in the document.\u201d We ran six frontier models on these with zero abstention coaching and they asserted made up answers 11% to 44% of the time. The full per model table is on the dataset card with raw logs and API errors disclosed.<\/p>\n
What\u2019s in it: 2,889 certified absence rows, 3,088 span verified extractive QA rows, a 127K token packed long context task set, and a split minted only from SEC filings dated after every major model\u2019s training cutoff. That fresh split regenerates monthly, so it stays impossible to have trained on, by construction.<\/p>\n
Every row carries a certificate you can re-check yourself in a few lines of python, the audit snippet is on the card. When our own audits flag something, like extractive answers that are guessable from world knowledge (about 1.6% of them), we label it instead of quietly deleting it.<\/p>\n
Also worth knowing before you trust us: a reviewer caught one of our splits being weaker than claimed this week. We re-audited every row the same night, withdrew the split with per row evidence committed to the repo, tightened the protocol, and reshipped only the rows that survive everything. The full trail is in the audits folder, judge for yourself.<\/p>\n
License CC BY 4.0. Generation was an Apache 2.0 open weight model on our own hardware, the claim is the verification layer, not the generation. Held out versions never get published so they can\u2019t leak into training data. If anyone wants a sealed diagnostic run against their own model or domain (25 items, free, about a day), contact is on the card.<\/p>\n
https:\/\/huggingface.co\/datasets\/SovNodeAI\/certified-document-qa<\/a><\/p>\n<\/div>\n submitted by \/u\/Spirited_Archer1855 <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
