PatchBench: Evaluating AI Agents for Vulnerability Patching
AI agents have recently demonstrated strong performance in automated vulnerability patching. However, existing evaluations often validate a patch only by testing whether the provided Proof-of-Concept (PoC) input still triggers a crash. This leaves two key threats to validity: agents may reproduce memorized historical developer patches, or they may generate surface-level fixes that only suppress the reported crash. We study these concerns for C/C++ vulnerability patching. We introduce a patch similarity metric to detect memorized patches. On average, 25% of the agent patches exhibit substanti
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 60%Microsoft patches record number of security vulnerabilities, citing its use of AI →
- PossiblePossibly related (embedding) · 60%Patch Now! Hackers targeting Critical vulnerability in AI engineering platform MLflow - Cyber Daily →
- FuzzySimilar title/name (fuzzy) · 87%NirDiamant/GenAI_Agents →
“Fuzzy title match (0.94): “PatchBench: Evaluating AI Agents for Vulnerability Patching” ≈ “NirDiamant/GenAI_Agents””
- FuzzySimilar title/name (fuzzy) · 84%Unity-Technologies/ml-agents →
“Fuzzy title match (0.92): “PatchBench: Evaluating AI Agents for Vulnerability Patching” ≈ “Unity-Technologies/ml-agents””
- FuzzySimilar title/name (fuzzy) · 59%datawhalechina/hello-agents →
“Fuzzy title match (0.73): “PatchBench: Evaluating AI Agents for Vulnerability Patching” ≈ “datawhalechina/hello-agents””
- FuzzySimilar title/name (fuzzy) · 59%jnMetaCode/agency-agents-zh →
“Fuzzy title match (0.73): “PatchBench: Evaluating AI Agents for Vulnerability Patching” ≈ “jnMetaCode/agency-agents-zh””
- FuzzySimilar title/name (fuzzy) · 59%e2b-dev/awesome-ai-agents →
“Fuzzy title match (0.73): “PatchBench: Evaluating AI Agents for Vulnerability Patching” ≈ “e2b-dev/awesome-ai-agents””
- PossiblePossibly related (embedding) · 57%Why this month's Microsoft patch release is a doozy →