paperarXivTrust 82 · PrimaryPublished 3d agoLive · 2d ago

Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias

Existing studies of LLM-as-judge scoring bias work predominantly at the input-output level: they perturb inputs, measure score deltas, and propose prompt-level mitigations. We argue that the same biases admit a representation-level account in the judge's hidden state, complementary to the input-output view and operationally useful in ways it does not afford. We report three findings, across seven judges, seven bias types, and nine benchmarks. Geometry: baseline judging inputs occupy a tight activation manifold while biased inputs are displaced along a low-dimensional, type-specific subspace th

Lineage graph

Paper → model → repo connections mined from source citations (Tier-1 exact match).

Why these links exist

Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.

PossiblePossibly related (embedding) · 55%I had 55 LLMs blind-grade each other (22k judgments, all open). Every model family with enough data is biased toward its own siblings. Qwen judges favor Qwen by ~0.9 points. Mistral penalizes its own by ~1.0. →
PossiblePossibly related (embedding) · 48%variii/llm-as-a-judge →
LinkedLinked via arxiv author · 85%Zixiang Xu →
“Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias”
LinkedLinked via arxiv author · 85%Sixian Li →
“Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias”
LinkedLinked via arxiv author · 85%Huaxing Liu →
“Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias”
LinkedLinked via arxiv author · 85%Zhixiang Wang →
“Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias”
LinkedLinked via arxiv author · 85%Shuai Li →
“Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias”
LinkedLinked via arxiv author · 85%Zirui Song →
“Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias”

Covers

newsI had 55 LLMs blind-grade each other (22k judgments, all open). Every model family with enough data is biased toward its own siblings. Qwen judges favor Qwen by ~0.9 points. Mistral penalizes its own by ~1.0.

Implements

repovariii/llm-as-a-judge

authored (incoming)

personZixiang Xu personSixian Li personHuaxing Liu personZhixiang Wang personShuai Li personZirui Song personXiuying Chen

Implements (incoming)

repobytedance/deer-flow repomodular/modular reporay-project/ray

Related across the graph

repomodular/modular reporay-project/ray personShuai Li newsI had 55 LLMs blind-grade each other (22k judgments, all open). Every model family with enough data is biased toward its own siblings. Qwen judges favor Qwen by ~0.9 points. Mistral penalizes its own by ~1.0.personSixian Li repobytedance/deer-flow personHuaxing Liu personZhixiang Wang personZirui Song repovariii/llm-as-a-judge personZixiang Xu personXiuying Chen

Topics

cs.AI