Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction
Rubric-based evaluation is a promising approach for assessing open-ended outputs from LLM-based research agents, particularly in paper reproduction, where direct paper-to-repository comparison is prone to hallucination. However, constructing paper-specific rubrics requires substantial expert effort, limiting the scalability of benchmarks such as PaperBench. In this work, we present, to our knowledge, the first systematic meta-evaluation of LLM-generated rubrics for paper reproduction. We reformulate rubrics into a checklist-style format and evaluate four generation settings across two backbone
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 54%OskarsEzerins/llm-benchmarks →
- PossiblePossibly related (embedding) · 51%Agent-Field/pr-af →
- PossiblePossibly related (embedding) · 51%YerbaPage/Awesome-Repo-Level-Code-Generation →
- PossiblePossibly related (embedding) · 50%Giskard-AI/giskard-oss →
- PossiblePossibly related (embedding) · 49%federicodeponte/opendraft →
- LinkedLinked via arxiv author · 85%Hanhua Hong →
“Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction”
- LinkedLinked via arxiv author · 85%Yizhi Li →
“Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction”
- LinkedLinked via arxiv author · 85%Jiaoyan Chen →
“Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction”