A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks
Multiple-choice medical benchmarks are increasingly saturated, and recent rubric-based evaluations such as HealthBench have shown that open-ended clinical performance is far from solved - its "Hard" subset top score remains 32%. We present a small, deliberately difficult evaluation dataset of five clinician-authored clinical scenarios spanning four specialties (anaesthesia, internal/family medicine, emergency medicine, and obstetrics), each accompanied by an atomic, weighted, MECE rubric (25-62 criteria per task; 184 criteria total) authored from a clinician-drafted golden answer. We evaluate
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 45%New benchmark exposes reasoning gaps in top models →
- LinkedLinked via arxiv author · 85%Samiha A. Ismail →
“A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks”
- LinkedLinked via arxiv author · 85%Fan X. Chen →
“A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks”
- LinkedLinked via arxiv author · 85%Ali Merali →
“A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks”
- FuzzySimilar title/name (fuzzy) · 59%rasbt/reasoning-from-scratch →
“Fuzzy title match (0.73): “A rubric-based controlled comparison of frontier language mo” ≈ “rasbt/reasoning-from-scratch””
- PossiblePossibly related (embedding) · 48%Clinician Use of a General-Purpose Large Language Model in Hospital Medicine: A Mixed-Methods Pilot Study - Cureus →
- PossiblePossibly related (embedding) · 48%A Psychometric Comparison of Faculty-Authored and Large Language Model-Generated Multiple-Choice Questions in Endodontics - Cureus →