I’ve been experimenting with data-efficient LLM training as part of a project I’m calling Oren, focused on entropy-based dataset filtering.<\/h1>\n
The philosophy behind this emerged from knowledge distillation pipelines, where student models basically inherit the same limitations of intelligence as the teacher models have. Thus, the goal of Oren is to change LLM training completely \u2013 from the current frontier approach of rapidly upscaling in compute and GPU hours to a new strategy: optimizing training datasets for smaller, smarter models.<\/p>\n
The experimentation setup: two identical 100M-parameter language models.<\/p>\n
- \n
- Model A:<\/strong> trained on 700M raw tokens<\/li>\n
- Model B:<\/strong> trained on the top 70% of samples (500M tokens) selected via entropy-based filtering<\/li>\n<\/ul>\n
Result:<\/strong> Model B matched Model A in performance, while using 30% less data, time, and compute. No architecture or hyperparameter changes.<\/p>\n
Open-source models:<\/p>\n
\ud83e\udd17 Model A – Raw (700M tokens)<\/a><\/p>\n
\ud83e\udd17 Model B – Filtered (500M tokens)<\/a><\/p>\n
Full documentation:<\/p>\n
\ud83d\udc7eGitHub Repository<\/a><\/p>\n
I’d love feedback, especially on how to generalize this into a reusable pipeline that can be directly applied onto LLMs before training and\/or fine-tuning\u2013I’m currently thinking of a multi-agent system, with each agent being a SLM trained on a subdomain (i.e., coding, math, science), each with their own scoring metrics. Would love feedback from anyone here who has tried entropy or loss-based filtering and possibly even scaled it<\/p>\n<\/div>\n
submitted by \/u\/Jolly-Act9349 <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p> - Model B:<\/strong> trained on the top 70% of samples (500M tokens) selected via entropy-based filtering<\/li>\n<\/ul>\n
