Posterior Distillation Index: A New Metric for Verifying Agent Skills
A new arXiv paper (2605.09192) introduces the Posterior Distillation Index (PDI), a trajectory-level metric for assessing the quality of agent skills distilled from procedural documents. The authors argue that existing skill generation methods rely on preference logs rather than direct environment interaction, leading to negligible or degraded gains. They propose SPARK (Structured Pipelines for Autonomous Runnable tasKs and sKill generation), a system that generates environment-verified trajectories to compute PDI, enabling robust skill verification grounded in empirical evidence.
Key facts
- arXiv paper 2605.09192 introduces Posterior Distillation Index (PDI)
- PDI is a trajectory-level metric for skill verification
- Existing methods rely on preference logs, not environment interaction
- SPARK generates environment-verified trajectories
- Skill quality is difficult to assess without environment-grounded verification
- Robust skills should be posterior-based, distilled from empirical interaction
- Paper identifies a timing bottleneck in skill generation
- SPARK stands for Structured Pipelines for Autonomous Runnable tasKs and sKill generation
Entities
Institutions
- arXiv