Study Reveals Agent Skills Can Cause Failures in LLM Agents
A new study on arXiv (2608.11888) looks into the downsides of using skills in large language model (LLM) agents. These skills are designed to improve LLMs by providing reusable guidance for tasks like planning and problem-solving. However, past studies have shown mixed results—some skills boost success rates, while others can increase token usage, extend execution times, or even lower success rates. This research closely examines failures linked to these skills, identifying 307 instances of skill-related failures, which include 125 functional and 182 efficiency issues. The researchers suggest a framework for analyzing the impact of skills by comparing outcomes with and without them, highlighting the need for careful evaluation before implementing these skills in LLMs. You can find the paper on arXiv at 2608.11888.
Key facts
- The study is titled 'Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents'.
- It is published on arXiv with identifier 2608.11888.
- Agent skills are used to extend LLM agents with reusable guidance.
- The study introduces a differential analysis framework to attribute failures to specific skills.
- The framework compares skill-guided runs against no-skill or semantically matched reference runs.
- The framework was instantiated on SkillsBench and SWE-Skills-Bench.
- The study found 307 skill-induced failures, including 125 functional failures and 182 efficiency regressions.
- The research highlights that skills can increase token use and execution time, and reduce success rates.
Entities
Institutions
- arXiv