{"id":40757,"date":"2026-05-01T21:23:36","date_gmt":"2026-05-01T19:23:36","guid":{"rendered":"https:\/\/www.graviton.at\/letterswaplibrary\/built-a-cli-to-clean-github-repos-into-llm-training-data-is-manual-cleanup-a-real-bottleneck-for-you\/"},"modified":"2026-05-01T21:23:36","modified_gmt":"2026-05-01T19:23:36","slug":"built-a-cli-to-clean-github-repos-into-llm-training-data-is-manual-cleanup-a-real-bottleneck-for-you","status":"publish","type":"post","link":"https:\/\/www.graviton.at\/letterswaplibrary\/built-a-cli-to-clean-github-repos-into-llm-training-data-is-manual-cleanup-a-real-bottleneck-for-you\/","title":{"rendered":"Built A CLI To Clean GitHub Repos Into LLM Training Data.. Is Manual Cleanup A Real Bottleneck For You?"},"content":{"rendered":"

<\/p>\n

\n

Every time I try to use GitHub repos for LLM training, I lose hours cleaning junk like .git files, lock files, minified JS, generated code, binaries mixed with real source.<\/p>\n

Public datasets like The Stack are great for general pretraining. But if you want a model on a specific stack or curated repos, you end up building the dataset (and the cleanup pipeline) yourself. So I built a CLI tool called RepoCurator to make that step reusable.<\/p>\n

What it does:<\/p>\n

– Clones repos (shallow for speed)<\/p>\n

– Filters noise using rules<\/p>\n

– Scores files (0.0\u20131.0) based on usefulness<\/p>\n

– Exports clean, per-file datasets (JSON\/TXT)<\/p>\n

Still early trying to validate if this is a real problem. If it resonates, register interest on the page. It helps me decide whether to keep building<\/p>\n

Question:<\/p>\n

How are you currently cleaning repos before using them for training or analysis?<\/p>\n<\/div>\n

submitted by \/u\/Ok_Rub3312 <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>