We’ve been building a public quality standard for AI training data \u2014 same idea as Moody’s for bonds \u2014 and the free audit tool is now open to anyone. No account needed.<\/p>\n
What you get if you paste a HuggingFace dataset URL at https:\/\/labelsets.ai\/rate<\/a><\/p>\n \u2022 A 19-dimension quality score (structural, annotation, training-fit, compliance)<\/p>\n \u2022 7-oracle consensus across 5 algorithm families with Cohen + Fleiss \u03ba agreement reporting<\/p>\n \u2022 95% Wilson confidence intervals on rate-based dimensions<\/p>\n \u2022 90% conformal prediction interval on downstream model F1 (Vovk 2005 \/ Romano 2019)<\/p>\n \u2022 Contamination flags against 40+ public evals \u2014 MMLU, HumanEval, GSM8K, MedQA, LegalBench, SQuAD, ARC, TruthfulQA,<\/p>\n etc.<\/p>\n \u2022 An Ed25519-signed cert verifiable offline against our public key (fingerprint aa4c070af907e2ea)<\/p>\n Methodology paper is published open CC BY 4.0 (19 pages, peer-review ready) at labelsets.ai\/paper<\/a> \u2014 fork it, reimplement it, write a paper that disagrees with us.<\/p>\n The free \/rate audit produces a JSON cert. The hosted PDF + permalink + embeddable badge are paid ($49 procurement \/ $149 pro), but the underlying score is the same.<\/p>\n Built deliberately so verification works at FedRAMP-restricted shops \u2014 public API at GET \/api\/verify-lqs-cert\/:hash, no auth required, or run crypto.verify() against the Ed25519 public key locally.<\/p>\n Curious what people here think of the dimension list. Happy to defend any of the 19 or kill the ones that don’t carry weight.<\/p>\n<\/div>\n submitted by \/u\/plomii <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
