When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed. We treat this evaluator-replacement ambiguity as a measurement-validity problem. Across four judgment datasets, we compare two upgrade paths available in practice: scaling Qwen3 dense judges from 1.7B to 32B parameters and moving across MiniMax M2-M2.7 released APIs. The main pattern is that judge upgrades are not interchangeable: only Qwen3 1.7B to 4B gives a robust adjacent gain, while MiniMax adjacent releases do not. Stronger judges reduce but do not remove position and ve
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 51%Evaluate a model properly →
- PossiblePossibly related (embedding) · 49%eval-harness-plus →
- PossiblePossibly related (embedding) · 47%variii/llm-as-a-judge →
- LinkedLinked via arxiv author · 85%Zongyou Yang →
“When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability”
- LinkedLinked via arxiv author · 85%Yinghan Hou →
“When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability”
- LinkedLinked via arxiv author · 85%Xiaokun Yang →
“When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability”
- PossiblePossibly related (embedding) · 49%DaoyuanLi2816/pairjudge →