Read original ↗
paperarXivTrust 82 · PrimaryPublished 11d agoLive · 8d ago

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

Language-model judges now gate training data, score generations, and drive leaderboards. The judge is then a measurement instrument, resting on one rarely stated assumption: the same request, sent to the same model name, reads the same tomorrow. We audited that assumption in two preregistered campaigns with every threshold fixed in advance; neither got past validating its instrument. Across 52,988 audited request attempts, same-window repeat rankings agreed at Spearman 0.400 against a required 0.90, and byte-identical next-day replays agreed at 0.78 against a required 0.99, each time with the

Lineage graph

Paper → model → repo connections mined from source citations (Tier-1 exact match).

Why these links exist

Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.

Explains

tutorialEvaluate a model properly

Covers

newsCan a MUD evaluate LLMs? A $99 proof of conceptnewsAt matched bits, the downstream winner between two codes flips when you change the consumer — while reconstruction error says they're identical. 12-domain study, pre-registered, negative results included` [R]

Implements (incoming)

repopatchy631/ai-engineering-hubrepostas00/ml-engineeringrepocobusgreyling/loop-engineeringrepoNirDiamant/Prompt_Engineeringreporohitg00/ai-engineering-from-scratch

authored (incoming)

personHaoyaun ZhupersonHongjie Zhang

Covers (incoming)

newsAgent Evaluation Metric for multi-turn conversations

Related across the graph

newsCan a MUD evaluate LLMs? A $99 proof of conceptpersonHongjie Zhangrepostas00/ml-engineeringrepopatchy631/ai-engineering-hubreporohitg00/ai-engineering-from-scratchnewsAt matched bits, the downstream winner between two codes flips when you change the consumer — while reconstruction error says they're identical. 12-domain study, pre-registered, negative results included` [R]newsAgent Evaluation Metric for multi-turn conversationsrepoNirDiamant/Prompt_EngineeringpersonHaoyaun ZhututorialEvaluate a model properlyrepocobusgreyling/loop-engineering

Topics

cs.AI