Second Guess: A Lightweight Method for Uncertainty Detection in Small Language Models
A new technique called Second Guess has been introduced by researchers to enhance abstention in multiple-choice question answering (MCQA) for small language models (SLMs) without requiring parameters. This approach is based on the observation that models which are confident in their answers tend to select them consistently, whereas those that are uncertain exhibit erratic behavior when presented with an 'I don't know' option. Tested on four open models (ranging from 2B to 8B parameters) and four benchmarks, Second Guess results in a composite risk reduction of 10.81%. It also shows an 8% improvement on fine-tuned models where entropy-based methods falter, particularly benefiting lower-performing models. This method tackles the significant issue of SLMs providing confident yet incorrect responses due to their operational limitations.
Key facts
- Second Guess is a lightweight, parameter-free prompting technique for abstention in MCQA.
- It is designed for small language models (SLMs) with 2B-8B parameters.
- The method detects uncertainty by observing answer stability when an 'I don't know' option is added.
- Evaluated on four open models and four benchmarks.
- Achieves highest composite risk improvement of 10.81%.
- Maintains 8% composite risk improvement on fine-tuned models.
- Entropy-based methods degrade on fine-tuned models.
- Improves most for lower-performing models.
Entities
Institutions
- arXiv