Game Theoretic Free Energy Analysis Reveals Higher Order Redundancy in LLM Attention Heads
A recent study published on arXiv (2605.09515) utilizes the Game Theoretic Free Energy Principle (GTFEP) to investigate multihead attention in extensive language models. This approach considers attention heads as rational agents that aim to minimize variational free energy, with their collective actions adhering to a Gibbs distribution across coalition frameworks. By employing a manageable approximation with a uniform prior and deterministic dynamics, coalition free energy simplifies to the joint Shannon entropy of discretized outputs from the heads. Pairwise dividends equate to mutual information (which is nonnegative), whereas triple dividends relate to interaction information, which can be negative. Experiments conducted on BERT, GPT2, and Llama using GSM8K consistently indicate negative triple dividends, highlighting higher-order redundancy. Additionally, the paper presents the Nash FEP correspondence.
Key facts
- Paper applies Game Theoretic Free Energy Principle to multihead attention in LLMs
- Framework treats attention heads as bounded rational agents minimizing variational free energy
- Collective behavior follows Gibbs distribution over coalition structures
- Tractable approximation uses uniform prior and deterministic dynamics
- Coalition free energy reduces to joint Shannon entropy of discretized head outputs
- Pairwise dividends become mutual information (nonnegative)
- Triple dividends correspond to interaction information and can be negative
- Experiments on BERT, GPT2, and Llama with GSM8K show consistently negative triple dividends
- Negative triple dividends indicate higher order redundancy
- Paper introduces Nash FEP correspondence
Entities
Institutions
- arXiv