Hi everyone! I wanted to share a benchmark we developed for testing our LLM-based symptom checker app. We built this because existing static benchmarks (like MedQA, PubMedQA) didn\u2019t fully capture the real-world utility of our app. With no suitable benchmark available, we created our own and are open-sourcing it in the spirit of transparency.<\/p>\n
GitHub: https:\/\/github.com\/medaks\/symptomcheck-bench<\/a><\/p>\n Quick Summary:<\/strong> <\/p>\n We call it SymptomCheck Bench<\/strong> because it tests the core functionality of symptom checker apps\u2014extracting symptoms through text-based conversations and generating possible diagnoses. It’s designed to evaluate how well an LLM-based agent can perform this task in a simulated setting.<\/p>\n The benchmark has three main components:<\/p>\n Patient Simulator<\/strong>: Responds to agent questions based on clinical vignettes. Symptom Checker Agent<\/strong>: Gathers information (limited to 12 questions) to form a diagnosis. Evaluator agent<\/strong>: Compares symptom checker diagnoses against the ground truth diagnosis. <\/p>\n Key Features:<\/p>\n 400 clinical vignettes from a study comparing commercial symptom checkers. Multiple LLM support (GPT series, Mistral, Claude, DeepSeek) Auto-evaluation system validated against human medical experts <\/p>\n We know it’s not perfect, but we believe it’s a step in the right direction for more realistic medical AI evaluation. Would love to hear your thoughts and suggestions for improvement!<\/p>\n<\/div>\n submitted by \/u\/Significant-Pair-275 <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
