Probe-Guided Training Improves LLM Tool Call Parameter Accuracy
A recent paper on arXiv (2608.03071) presents a benchmark that varies in difficulty alongside a training framework guided by probes, aimed at tackling the often-overlooked issue of parameter filling in large language model (LLM) tool calls. The researchers note that even the most advanced models struggle, completing less than 50% of tool calls accurately in areas like cloud networking, underscoring the urgent need for enhanced parameter precision. They found that a model's hidden state provides a robust signal for correctness, allowing a straightforward linear probe to predict the accuracy of generated parameter values. Their proposed framework includes two synergistic methods: probe-filtered bootstrapped training (PBT) and probe-guided decoding (PGD). Additionally, the paper introduces a benchmark to assess parameter filling across different complexity levels, indicating that utilizing hidden state signals can greatly enhance the reliability of LLM tool applications, particularly in real-world scenarios such as cloud networking.
Key facts
- Paper arXiv:2608.03071 focuses on parameter filling in LLM tool calls.
- Existing research has largely focused on tool selection and call ordering, not parameter accuracy.
- In cloud networking, frontier models correctly complete fewer than half of tool calls.
- A simple linear probe can predict parameter correctness from hidden states.
- Proposed framework includes probe-filtered bootstrapped training (PBT) and probe-guided decoding (PGD).
- A difficulty-graded benchmark is introduced for evaluating parameter filling.
- The paper is announced as a new arXiv preprint.
- The study is inspired by analyses showing LLM hidden states encode rich prediction information.
Entities
Institutions
- arXiv