Probes Detect Corrupted Context but Fail to Predict Errors in Language Models

A new study on arXiv (2608.07528) reveals a gap between how well linear probes can detect corruption in language models and their ability to predict errors. While these probes can accurately identify corrupt contexts, they fail to assess the accuracy of answers in multi-step arithmetic problems. The research, which included pre-registered hypotheses, found that models using structured confidence formats often settle on two similar error rates. Furthermore, the persistence of probes over multiple steps doesn't effectively differentiate between right and wrong answers, challenging the 'persistence beats peak' idea. This pattern is seen across different model types. The effectiveness of probe-based monitoring varies based on the model and error type, with branch-and-pick showing promise, especially with Llama-3.1-8B.