I’ve published a small text-classification dataset for support-ticket routing, together with the per-case answers of five models. It’s my own project.
What’s in it
- 125 fictional English support messages, each labelled with one of five teams:
billing,technical,account,sales, orneeds_clarification(the message doesn’t say enough to route). - Splits: 25 development cases and 100 held-out test cases, balanced at 25 per label overall.
- Per case: the expected label, a written rationale, tags, a category (clear, boundary, ambiguous, adversarial) and a difficulty.
- The label policy as a separate file, including precedence rules such as “route the immediate blocker, not the eventual goal”.
- 500 result rows: Jev, Clef, Clef Flash, OpenAI’s Decisions API and GPT-6 Luna with structured output on the 100 test cases, with each answer, status, response time and token counts, and per-label probabilities where the provider returns them.
How it was made
The messages are fictional, drafted with AI assistance under the written policy, then reviewed by two people. No real customer conversations were used.
Limits
- It is small and English only.
- It is balanced by design, so it is not representative of a real queue.
- The test cases are now public, so they are no longer a clean held-out set for future tuning.
Links
- Dataset (CC BY 4.0): huggingface.co/datasets/decisionmodelhub/support-routing
- Results page with a case-by-case explorer: decisionmodelhub.com/benchmarks/support-routing
Load it
from datasets import load_dataset cases = load_dataset("decisionmodelhub/support-routing", "cases") results = load_dataset("decisionmodelhub/support-routing", "results", split="test") If you work with support data: what would make a larger version more useful to you? Longer messages, realistic label imbalance, more languages?
submitted by /u/AncientComment2352
[link] [comments]
