Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents
Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 26%GreyDGL/PentestGPT →
“Possibly related via embedding similarity 0.56 (not asserted). Timestamp check: artifact slightly before paper (-3d).”
- FuzzySimilar title/name (fuzzy) · 87%NirDiamant/GenAI_Agents →
“Fuzzy title match (0.94): “Beyond Success Rate: Cost-Aware Evaluation of Offensive and ” ≈ “NirDiamant/GenAI_Agents””
- FuzzySimilar title/name (fuzzy) · 84%Unity-Technologies/ml-agents →
“Fuzzy title match (0.92): “Beyond Success Rate: Cost-Aware Evaluation of Offensive and ” ≈ “Unity-Technologies/ml-agents””
- FuzzySimilar title/name (fuzzy) · 59%datawhalechina/hello-agents →
“Fuzzy title match (0.73): “Beyond Success Rate: Cost-Aware Evaluation of Offensive and ” ≈ “datawhalechina/hello-agents””
- FuzzySimilar title/name (fuzzy) · 59%Eigenwise/atomic-agents →
“Fuzzy title match (0.73): “Beyond Success Rate: Cost-Aware Evaluation of Offensive and ” ≈ “Eigenwise/atomic-agents””
- FuzzySimilar title/name (fuzzy) · 59%TauricResearch/TradingAgents →
“Fuzzy title match (0.73): “Beyond Success Rate: Cost-Aware Evaluation of Offensive and ” ≈ “TauricResearch/TradingAgents””
- LinkedLinked via arxiv author · 85%Paul Kassianik →
“Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents”
- LinkedLinked via arxiv author · 85%Blaine Nelson →
“Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents”