LLM agent benchmarks like \u03c4-bench ask what agents can<\/em> do. Real deployment asks something harder: do they know when they<\/strong> shouldn\u2019t<\/em><\/strong> act?<\/strong><\/p>\n CAR-bench (<\/strong>https:\/\/arxiv.org\/abs\/2601.22027<\/a>)<\/strong>, a benchmark for automotive voice assistants with domain-specific policies, evaluates three critical LLM Agent capabilities:<\/p>\n 1\ufe0f\u20e3 Can they complete multi-step requests? Three targeted task types:<\/p>\n \u2192 Base<\/strong> (100 tasks): Multi-step task completion tested in a realistic evaluation sandbox: What was found:<\/strong> Completion over compliance.<\/em><\/p>\n SOTA model (Claude-Opus-4.5): only 52% consistent success.<\/p>\n Hallucination: non-thinking models fabricate more often; thinking models improve but plateau at 60%.<\/p>\n Disambiguation: no model exceeds 50% consistent pass rate. GPT-5 succeeds 68% occasionally, but only 36% consistently.<\/p>\n The gap between “works sometimes” and “works reliably” is where deployment fails.<\/p>\n \ud83e\udd16 Curious how to build an agent that beats 54%?<\/p>\n \ud83d\udcc4 Read the Paper: https:\/\/arxiv.org\/abs\/2601.22027<\/a><\/p>\n \ud83d\udcbb Run the Code & benchmark: https:\/\/github.com\/CAR-bench\/car-bench<\/a><\/p>\n We’re the authors – happy to answer questions!<\/strong><\/p>\n<\/div>\n submitted by \/u\/Frosty_Ad_6236 <\/a>
2\ufe0f\u20e3 Do they admit limits\u2014or fabricate capabilities?
3\ufe0f\u20e3 Do they clarify ambiguity\u2014or just guess?<\/p>\n
\u2192 Hallucination<\/strong> (90 tasks): Admit limits vs. fabricate
\u2192 Disambiguation<\/strong> (50 tasks): Clarify vs. guess<\/p>\n
58 tools \u00b7 19 domain policies \u00b7 48 cities \u00b7 130K POIs \u00b7 1.7M routes \u00b7 multi-turn interactions.<\/p>\n\n
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
