Most dataset quality labels I’ve seen are a single score (accuracy, or “is_valid: true”). After building three reasoning datasets for LLM fine-tuning (legal, clinical, financial) I kept hitting cases where a single score hid the actual problem \u2014 e.g., an answer that was factually correct but cited a nonexistent case, or one with perfect citations but a broken reasoning chain.<\/p>\n
So I broke quality into 7 dimensions, scored per-example:<\/strong><\/p>\n Correctness \u2014 does the conclusion match ground truth?<\/p>\n<\/li>\n Reasoning coherence \u2014 does each step follow from the previous?<\/p>\n<\/li>\n Citation accuracy \u2014 every reference verified against source?<\/p>\n<\/li>\n Completeness \u2014 are all required fields populated?<\/p>\n<\/li>\n Factual grounding \u2014 any hallucinated facts?<\/p>\n<\/li>\n Consistency \u2014 are labels applied the same way across the corpus?<\/p>\n<\/li>\n Reproducibility \u2014 can the conclusion be re-derived from the rule\/inputs alone?<\/p>\n<\/li>\n<\/ol>\n Each dimension gets 0.0\u20131.0. Final score is the geometric mean (one bad dimension should tank the example, not average out). Low-scoring examples are kept in the corpus but flagged in metadata so downstream users can filter them.<\/p>\n What surprised me during scoring:<\/strong><\/p>\n – ~18% of GPT-4 generated legal analyses had fabricated citations that looked real (wrong year, wrong court, right-ish case name)<\/p>\n – Reasoning coherence and citation accuracy were almost uncorrelated \u2014 you can have one without the other<\/p>\n – Consistency (dimension 6) was the hardest to measure and the most valuable once I did \u2014 it surfaced a whole class of “label drift” where mid-corpus annotation standards had shifted<\/p>\n Applied to:<\/strong><\/p>\n – 445 US appellate legal reasoning examples (median score 0.92)<\/p>\n – 493 clinical reasoning traces (median 0.88)<\/p>\n – 1,000 financial routing\/classification examples (median 0.94)<\/p>\n Full methodology writeup: https:\/\/labelsets.ai\/lqs-methodology<\/a><\/p>\n Genuinely curious:<\/strong><\/p>\n – Has anyone tried something similar with more\/fewer dimensions?<\/p>\n – Is geometric mean the right aggregation, or does anyone use a weighted model?<\/p>\n – For reasoning datasets specifically, which dimensions are you most suspicious of when evaluating external data before buying\/using it?<\/p>\n Happy to go deeper on any dimension in the comments.<\/em><\/strong><\/p>\n<\/div>\n submitted by \/u\/plomii <\/a> \n
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
