Semia: Static Auditor for LLM Agent Skills via Constraint-Guided Synthesis
A new tool called Semia has been developed to assess the abilities of agents powered by LLMs. In this context, an agent skill is essentially a package that equips an agent with various functions, like reading emails or signing transactions on the blockchain. Each skill is made up of two parts: one part lays out clear interfaces that can be executed, while the other part consists of prose that describes the circumstances under which these interfaces activate. Traditional tools often miss the prose aspect, while LLM-based tools struggle to confirm if a risky input leads to a significant issue. Semia converts skills into the Skill Description Language (SDL), which captures the necessary actions, prose conditions, and checkpoints for human review. The main goal is to ensure the fact base remains both structurally sound and semantically true to the original prose. This research can be found in a paper on arXiv, ID 2605.00314.
Key facts
- Semia is a static auditor for agent skills.
- Agent skills equip LLM-driven agents with capabilities like reading email, executing shell commands, or signing blockchain transactions.
- Each skill is a hybrid artifact with a structured half and a prose half.
- Conventional static analyzers ignore the prose half.
- LLM-based tools cannot reproducibly prove tainted input reaches a high-impact sink.
- Semia uses the Skill Description Language (SDL), a Datalog fact base.
- SDL captures LLM-triggered actions, prose-defined conditions, and human-in-the-loop checkpoints.
- The paper is on arXiv with ID 2605.00314.
Entities
Institutions
- arXiv