I Built A Gated Dataset Pipeline For Fine-tuning Small Local Models, Here’s How The Checks Work

I’ve been deep-fine-tuning small local models (BitNet 1.58, Qwen 1.7B/4B, Gemma-4) into a real on-device assistant with actual tool-calling. The hard part was never the training loop, it’s the data. Scraped

ShareGPT dumps collapse small models into formulaic slop and teach tool syntax that doesn’t match your runtime. And for full/deep fine-tuning (not LoRA), bad data is fatal: a small model trained on redundant, single-voice data monocultures hard.

So I built a pipeline where every example has to survive a set of hard gates before it’s allowed in.

Sharing how it works, because I rarely see people talk about the controls, only the volume.

The core: hand-written “gold seed” → multi-teacher expansion

– The seed is hand-written, one example at a time, in a neutral model-agnostic format ({messages, tools}).

– It’s rendered per dialect: ChatML for Qwen/BitNet, native Gemma tool-call format for Gemma. Same data, correct syntax for each target.

– From a curated seed it expands to hundreds of thousands of examples on demand — volume comes from multiple teacher models of different families (anti style-collapse), each example tagged with the teacher that produced it. You scale to the size you actually need.

The gates (this is the value)

– Anti-formula: blocks over-used openers/closers at ingest + global frequency caps; any phrase repeated too often anywhere gets flagged. This is what stops a full fine-tune from collapsing into one voice.

– Semantic dedup (BGE-M3): near-duplicate detection, not byte-match. On one 9k combinatorial corpus it found 43% near-duplicates, paraphrases expansion would’ve amplified. Keeps 1 per cluster, with a coverage guard that never drops the sole example teaching a capability.

– Flow gate: multi-turn tool-call integrity: every assistant tool_call is followed by exactly its results, no orphans, no pending calls, ends with a real answer.

– Dialect gate: every tool-call is round-tripped through each target model’s syntax and rejected if it doesn’t parse back identical. train == runtime, guaranteed.

– Coverage: every tool taught above threshold; catalog-sync rejects phantom (hallucinated) and untaught tools.

– Vision routing: vision examples go only to vision-capable models; text-only models never see image context they can’t use at inference.

– Judge routing — teacher outputs that pass go to SFT; the ones that fail become KTO negatives (preference signal, not trash).

Built for deep fine-tuning, at scale

The whole point of the gates is that you can full-fine-tune a small model on this without it collapsing and expand a small hand-verified seed to 100k+ examples while keeping every one of those checks green.

Tool-calling, multi-turn, charts/tables/HTML, per-model vision, reasoning, all gate-verified, neutral format so you render to your own dialect.

What it trains

Liara — a local personal AI with 24 real tools (email, calendar, files, notes, web, weather, charts), english and italian-first + multilingual, that resists prompt-injection while still handling legitimate reasoning-test / structured-output prompts (the distinction most classifiers get wrong).

– Liara app: https://nothumanallowed.com/local

– Tooling / code: https://github.com/adoslabsproject-gif/Liara-toolkit