New Framework Addresses Repetition in AI Language Models
Researchers have proposed a principled framework to combat attention collapse in autoregressive language models, a problem causing repetitive token loops. The method uses an adjacent-conditional probability construction to compare a token's observed frequency against its corpus prior, producing a self-normalizing penalty ratio that requires no ad hoc standardization. This correction is applied via a closed-form logit offset with zero approximation error and accumulated into a frozen output-layer bias through exponential moving average. The approach can repair already collapsed models without intrusive modifications to standard training procedures.
Key facts
- Attention collapse causes repetitive token loops in autoregressive language models.
- Existing decoding-time heuristics fail to address the root cause.
- The framework penalizes anomalous confidence from collapsed generation patterns.
- It uses an adjacent-conditional probability construction comparing observed frequency to corpus prior.
- The self-normalizing penalty ratio R = f(m,n,p)/f(np,n,p) requires no ad hoc standardization.
- The correction is isolated from the loss gradient.
- It is accumulated into a frozen output-layer bias via exponential moving average.
- The method can repair already collapsed models without intrusive modifications.
Entities
—