Hey all,<\/p>\n
I’m building an LLM router (a system that routes each incoming prompt to the cheapest model likely to pass, rather than always sending everything to GPT-4). The core idea: if a prompt is simple enough for Mistral-7B, why pay for GPT-4? <\/p>\n
I\u2019m currently using the RouterBench<\/a> dataset a lot. These kinds of data are incredibly valuable because you get multiple model outputs for the exact same prompts, plus metadata like cost\/quality, which makes it much easier to experiment with routing strategies and selection policies.<\/p>\n I\u2019m wondering: are there other public datasets or benchmarks that provide:<\/p>\n They don\u2019t have to be as big or polished as RouterBench, but anything in this spirit (evaluation logs, comparison datasets, crowdsourced model outputs, etc.) would be super helpful. Links to GitHub, Hugging Face datasets, papers with released generations, or hosted eval platforms that export data are all welcome.<\/p>\n If you\u2019ve built your own multi-model eval logs and are open to sharing or partially anonymizing them, I\u2019d also love to hear about that.<\/p>\n Thanks!<\/p>\n<\/div>\n submitted by \/u\/Apart-Dot-973 <\/a> \n
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
