OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills
LLM-based agents leverage third-party skills to extend their capabilities in open-world scenarios. However, third-party skills can introduce extra security vulnerabilities, as seemingly harmless skills can contain latent safety risks that only emerge during actual execution. In this work, we conduct a systematic investigation into how well current agent systems recognize and avoid such risks. To support quantitative and qualitative evaluation, we construct OpenSkillRisk, a dedicated safety benchmark containing 263 risky skills collected from public skill marketplaces. We classify these skills
Lineage graph
Paper → model → repo connections mined from source citations (Tier-1 exact match).
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- FuzzySimilar title/name (fuzzy) · 59%AgentCore-8B →
“Fuzzy title match (0.73): “OpenSkillRisk: Benchmarking Agent Safety When Using Real-Wor” ≈ “AgentCore-8B””
- PossiblePossibly related (embedding) · 53%Safety and alignment in an era of long-horizon models →
- PossiblePossibly related (embedding) · 52%What does "Safe AI" look like? [D] →
- FuzzySimilar title/name (fuzzy) · 87%SWE-agent/SWE-agent →
“Fuzzy title match (0.94): “OpenSkillRisk: Benchmarking Agent Safety When Using Real-Wor” ≈ “SWE-agent/SWE-agent””
- FuzzySimilar title/name (fuzzy) · 87%zhayujie/CowAgent →
“Fuzzy title match (0.94): “OpenSkillRisk: Benchmarking Agent Safety When Using Real-Wor” ≈ “zhayujie/CowAgent””
- FuzzySimilar title/name (fuzzy) · 66%open-multi-agent/open-multi-agent →
“Fuzzy title match (0.78): “OpenSkillRisk: Benchmarking Agent Safety When Using Real-Wor” ≈ “open-multi-agent/open-multi-agent””
- FuzzyOverlapping authors or contributors · 62%modular/modular →
“Shared author/contributor keys: liu”
- FuzzySimilar title/name (fuzzy) · 59%NousResearch/hermes-agent →
“Fuzzy title match (0.73): “OpenSkillRisk: Benchmarking Agent Safety When Using Real-Wor” ≈ “NousResearch/hermes-agent””