I’ve been building an on-device assistant and, like anyone doing tool-calling SFT, kept shipping subtle junk into my dataset, hallucinated tool names, invented arguments, a tool-call with no result, the model over-calling a calculator for “2+2”, near-duplicate examples quietly collapsing my diversity. So I wrote a validator, and I figured it might save someone else the same headaches.
It runs entirely in your browser — you drop in your .jsonl and your tool catalog, and nothing is uploaded anywhere. Your dataset never leaves your machine (that mattered to me, and I assume to some of you too).
What it checks, per example, with a typed verdict (keep / KTO-negative / discard):
– structure & roles, and every tool_call name/args against your catalog (required present, no invented args, types)
– tool-flow (each call gets its result, no orphans, ends with a real answer)
– over-calling, verbosity/filler, prompt-injection defense
– lexical and semantic dedup (the embedding model runs locally too, via WASM)
– diversity (distinct n-grams) + a difficulty breakdown
It spits out clean.jsonl / kto-negatives.jsonl / rejected.jsonl so you can just use the output.
It’s free, no account, no catch. Link: nothumanallowed.com/tools/dataset-validator
It’s early and opinionated (the heuristics come from my own pipeline), so if it flags something it shouldn’t, or misses something it should, I’d genuinely like to hear it — happy to adjust. Hope it’s useful to someone.
submitted by /u/Key-Outcome-2927
[link] [comments]
