Retrieved but not ranked: surface-form bias in structural retrieval, from mathematics to agent trajectories
We evaluate embedding retrieval where surface form and meaning are pulled apart on purpose: retrieving items that share underlying structure but not wording, in two unrelated domains under one protocol, competition mathematics (MathNet-Retrieve; 500 queries, 117,088-item corpus) and embodied-agent trajectories (ALFWorld-derived; 118 queries, 336 trajectories). In mathematics the failure is complete: strict Hit@1 at the heaviest disguise tier is 0.0% for both production embedders (bootstrap 95% CI [0.0, 0.0]) while the correct item sits in the top 10 nearly always, and in 95.2 to 99.8% of misse
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- FuzzySimilar title/name (fuzzy) · 59%AgentCore-8B →
“Fuzzy title match (0.73): “Retrieved but not ranked: surface-form bias in structural re” ≈ “AgentCore-8B””
- PossiblePossibly related (embedding) · 49%Evaluating J-space entropy as an error predictor across 7 datasets on Qwen3-4B [R] →
- PossiblePossibly related (embedding) · 48%Embedding →
- FuzzySimilar title/name (fuzzy) · 87%SWE-agent/SWE-agent →
“Fuzzy title match (0.94): “Retrieved but not ranked: surface-form bias in structural re” ≈ “SWE-agent/SWE-agent””
- FuzzySimilar title/name (fuzzy) · 87%zhayujie/CowAgent →
“Fuzzy title match (0.94): “Retrieved but not ranked: surface-form bias in structural re” ≈ “zhayujie/CowAgent””
- FuzzySimilar title/name (fuzzy) · 59%NousResearch/hermes-agent →
“Fuzzy title match (0.73): “Retrieved but not ranked: surface-form bias in structural re” ≈ “NousResearch/hermes-agent””
- FuzzySimilar title/name (fuzzy) · 59%bojieli/ai-agent-book →
“Fuzzy title match (0.73): “Retrieved but not ranked: surface-form bias in structural re” ≈ “bojieli/ai-agent-book””
- FuzzySimilar title/name (fuzzy) · 59%2FastLabs/agent-squad →
“Fuzzy title match (0.73): “Retrieved but not ranked: surface-form bias in structural re” ≈ “2FastLabs/agent-squad””