On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification
Memory-based self-improving agents--those that learn from an online stream of tasks and improve over time by maintaining a textual memory bank--have shown great promise in recent literature. However, the reliability aspects of these methods have been critically overlooked. In this work, we conduct a comprehensive re-evaluation of two memory-based methods, broadening the scope of evaluation along two axes: (1) including multiple runs to quantify variance, and (2) randomly shuffling the tasks to investigate the effect of task order. Through these experiments, we make two observations that expose
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 56%We're building agents that can read millions of documents, but still forget a video they watched yesterday. →
- FuzzySimilar title/name (fuzzy) · 87%NirDiamant/GenAI_Agents →
“Fuzzy title match (0.94): “On the Fragility of Self-Improving Agents: Variance, Task Or” ≈ “NirDiamant/GenAI_Agents””
- FuzzySimilar title/name (fuzzy) · 84%Unity-Technologies/ml-agents →
“Fuzzy title match (0.92): “On the Fragility of Self-Improving Agents: Variance, Task Or” ≈ “Unity-Technologies/ml-agents””
- FuzzySimilar title/name (fuzzy) · 59%datawhalechina/hello-agents →
“Fuzzy title match (0.73): “On the Fragility of Self-Improving Agents: Variance, Task Or” ≈ “datawhalechina/hello-agents””
- FuzzySimilar title/name (fuzzy) · 59%jnMetaCode/agency-agents-zh →
“Fuzzy title match (0.73): “On the Fragility of Self-Improving Agents: Variance, Task Or” ≈ “jnMetaCode/agency-agents-zh””
- FuzzySimilar title/name (fuzzy) · 59%TauricResearch/TradingAgents →
“Fuzzy title match (0.73): “On the Fragility of Self-Improving Agents: Variance, Task Or” ≈ “TauricResearch/TradingAgents””
- LinkedLinked via arxiv author · 85%Qinyuan Ye →
“On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification”
- LinkedLinked via arxiv author · 85%Zhenyu Liu →
“On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification”