BeyondUncertainty: Using LLM Confidence to Route Retrieval

There's this innovative approach called BeyondUncertainty that uses the confidence levels from black-box language models to figure out when to look for outside information to answer questions. It starts by creating a temporary answer with a confidence score, then checks against a specific threshold based on validation data. If the confidence is low, it retrieves the top 5 results and formulates a new answer; if it's high, it simply provides the initial answer. In tests involving 27,000 policy cases across six QA benchmarks and three different model types, BeyondUncertainty recorded an average token-level F1 score of 0.483, which is better than the 0.467 from constant retrieval and 0.401 from no retrieval, while also reducing retrieval costs. This study addresses the problems of irrelevant information and high computational demands from random retrieval in retrieval-augmented generation.