Transformers' Symbolic Reasoning Limited by Representational Collapse

A new study on arXiv investigates why decoder-only transformer models struggle with abstract symbolic reasoning, specifically propositional logic problems given in-context. Previous work found that models fail to generalize to problems with unseen variable names, partly due to difficulty in copying unseen tokens. The new research reveals a key additional factor: representational collapse, where the unembeddings (last-layer weights) of unseen tokens converge to nearly identical vectors during training. This collapse makes it hard for the model to distinguish multiple unseen variables, especially when embedding and unembedding parameters are shared. The finding provides a mechanistic explanation for the effectiveness of heuristic interventions like 'active forgetting', which periodically resets token embeddings. The study combines theoretical analysis with empirical evidence, offering insights into the limitations of current transformer architectures in symbolic reasoning tasks.