newsAI NewsTrust 60Published 2mo agoLive · 3mo ago

New benchmark exposes reasoning gaps in top models

A harder evaluation suite shows even leading models struggle on multi-hop tasks.

benchmarks

Why these links exist

Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.

LinkedLinked via unknownRetrieval is underrated →
LinkedLinked via unknownNuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models →
LinkedLinked via unknownNorthwind AI →
LinkedLinked via unknownEvaluate a model properly →
LinkedLinked via unknownCOCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives →
LinkedLinked via unknownCan LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA →
LinkedLinked via unknownCognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction →
LinkedLinked via unknownThe Complexity Ceiling Benchmark: A Multi-Domain Evaluation of Sequential Reasoning Under Depth Scaling →

Covers

articleRetrieval is underrated

Covers (incoming)

paperNuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models companyNorthwind AI tutorialEvaluate a model properly paperCOCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives paperCan LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA paperCognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction paperThe Complexity Ceiling Benchmark: A Multi-Domain Evaluation of Sequential Reasoning Under Depth Scaling paperThinkProbe: Beyond Accuracy -- Structural Profiling of Open-Ended LLM Reasoning Traces via Non-Generative Thought Graphs paperTravel-Oriented Reasoning Large Language Model via Domain-Specific Knowledge Graphs paperQuery-Aware Spreading Activation for Multi-Hop Retrieval over Knowledge Graphs paperAre We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoning paperParametric Skills paperDoes Verbose Chain-of-Thought Really Help? In-Distribution Evidence that Content, Not Length, Matters paperBefore Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning paperGrounding LLM Reasoning under Incomplete Graph Evidence paperOn the Faithfulness of Post-Hoc Concept Bottleneck Models paperLearning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs paperFork-Think with Confidence paperModality-Driven Search with Holistic Trace Judging for ARC-AGI-2 paperThink in English, Answer in Korean: Efficient Adaptation of Multilingual Tool-Using Agents paperBridging the Gap Between Latent and Explicit Reasoning with Looped Transformers paperMessage Passing Enables Efficient Reasoning paperDiffusion-GR2: Diffusion Generative Reasoning Re-ranker repoZefan-Cai/R-KV paperTUDUM: A Turkish-Thinking Reasoning Pipeline for Qwen3.5-27B paperCheckRLM: Effective Knowledge-Thought Coherence Checking in Retrieval-Augmented Reasoning paperA rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks paperHNSW with Accuracy Guarantees Using Graph Spanners -- A Technical Report paperPluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages paperEstimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs paperTest-Time Scaling for Small VLMs on Multilingual Visual MCQ paperAdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification repos-JoL/open-reasoning paperPost-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape Pre-, Intra-, and Post-CoT Calibration paperAIMO Interpretability Challenge paperIs Your Model Thinking or Just Stagnating? PUMA: Diagnosing Reasoning Pathology via Phase-Momentum Alignment paperTwo-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness paperPoTRE: Test-Time Reasoning inspired by Cognitive Heterogeneity

Related across the graph

paperCheckRLM: Effective Knowledge-Thought Coherence Checking in Retrieval-Augmented Reasoning paperDiffusion-GR2: Diffusion Generative Reasoning Re-ranker paperCOCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives paperBefore Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning paperDoes Verbose Chain-of-Thought Really Help? In-Distribution Evidence that Content, Not Length, Matters paperEstimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs paperThe Complexity Ceiling Benchmark: A Multi-Domain Evaluation of Sequential Reasoning Under Depth Scaling paperPost-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape Pre-, Intra-, and Post-CoT Calibration paperHNSW with Accuracy Guarantees Using Graph Spanners -- A Technical Report companyNorthwind AI paperAre We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoning repos-JoL/open-reasoning paperMessage Passing Enables Efficient Reasoning paperLearning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs paperGrounding LLM Reasoning under Incomplete Graph Evidence paperAIMO Interpretability Challenge repoZefan-Cai/R-KV paperPoTRE: Test-Time Reasoning inspired by Cognitive Heterogeneity paperTravel-Oriented Reasoning Large Language Model via Domain-Specific Knowledge Graphs paperPluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages paperA rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks paperParametric Skills paperFork-Think with Confidence paperThink in English, Answer in Korean: Efficient Adaptation of Multilingual Tool-Using Agents paperQuery-Aware Spreading Activation for Multi-Hop Retrieval over Knowledge Graphs paperAdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification paperTwo-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness paperCan LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA paperThinkProbe: Beyond Accuracy -- Structural Profiling of Open-Ended LLM Reasoning Traces via Non-Generative Thought Graphs paperModality-Driven Search with Holistic Trace Judging for ARC-AGI-2 paperTUDUM: A Turkish-Thinking Reasoning Pipeline for Qwen3.5-27B tutorialEvaluate a model properly paperBridging the Gap Between Latent and Explicit Reasoning with Looped Transformers paperIs Your Model Thinking or Just Stagnating? PUMA: Diagnosing Reasoning Pathology via Phase-Momentum Alignment articleRetrieval is underrated paperNuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models paperTest-Time Scaling for Small VLMs on Multilingual Visual MCQ paperCognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction paperOn the Faithfulness of Post-Hoc Concept Bottleneck Models