Clinician-Level Agreement Without Clinical Caution: LLM Evaluator Limits in Medical AI Benchmarking
Open-response evaluation provides stronger clinical validity than multiple-choice benchmarks but creates a scoring bottleneck that motivates automated LLM-asa-Judge approaches. Whether such evaluators replicate clinical calibration and caution, however, remains untested. We introduce MedQADE, the first standardised open-response clinical benchmark for German, a major clinical language lacking native evaluation infrastructure, comprising 3,800 items annotated by ten practising physicians and nine Large Language Model (LLM) evaluators. The top-performing evaluator model, Gemini 3 Flash, reached
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- LinkedLinked via unknownTowards AI-augmented decision making in psychiatry →
- LinkedLinked via unknownIntroducing GeneBench-Pro →
- LinkedLinked via unknownEvaluate a model properly →
- LinkedLinked via arxiv author · 85%William Philipp →
“Clinician-Level Agreement Without Clinical Caution: LLM Evaluator Limits in Medical AI Benchmarking”
- LinkedLinked via arxiv author · 85%Finn Fassbender →
“Clinician-Level Agreement Without Clinical Caution: LLM Evaluator Limits in Medical AI Benchmarking”
- LinkedLinked via arxiv author · 85%Thorsten Langer →
“Clinician-Level Agreement Without Clinical Caution: LLM Evaluator Limits in Medical AI Benchmarking”
- LinkedLinked via arxiv author · 85%Martje Pauly →
“Clinician-Level Agreement Without Clinical Caution: LLM Evaluator Limits in Medical AI Benchmarking”
- LinkedLinked via arxiv author · 85%Rebecca Herzog →
“Clinician-Level Agreement Without Clinical Caution: LLM Evaluator Limits in Medical AI Benchmarking”