Instruction-Tuned LLMs Collapse to Single Output, Cannot Sample Distributions
A recent study published on arXiv (2607.25292) indicates that language models fine-tuned for instruction struggle to sample from various distributions, often defaulting to a single response. Researchers evaluated models from three distinct families with varying post-training methods and discovered that all instruction-tuned models underperformed on every task, whereas base models exhibited significantly fewer failures. Interestingly, a model that cannot sample from a distribution can still accurately characterize it in one instance, a phenomenon referred to as the KNOWS/DOES split. This issue is linked to a flawed sampling mechanism. These results raise concerns about the reliability of LLMs as substitutes for human survey participants in silicon sampling.
Key facts
- Instruction-tuned models collapse to a single output rather than sampling from distributions.
- The same persona on the same question returns the same answer on over half of items in a public-opinion benchmark.
- The collapse is sharp: internal probabilities concentrate on a single option.
- Instruction tuning amplifies the failure across three model families.
- Base models fail far less often than instruction-tuned models.
- The KNOWS/DOES split describes the gap between describing and sampling a distribution.
- The failure is traced to a degenerate sampling primitive.
- The study is published on arXiv with ID 2607.25292.
Entities
Institutions
- arXiv