Deep Learning Architectures Fail on Indexing Primitive Without Masking

A recent study published on arXiv (2607.22361) examines information bottlenecks in contemporary deep-learning models, such as RNNs, softmax transformers, linear-attention transformers, and state-space models, using the indexing primitive as a framework. In this context, the input comprises n bits and an integer i (the index), with the output being the value of the i-th bit. The researchers present the concept of causal complexity for masked architectures, revealing that those with low causal complexity are unable to resolve the indexing primitive in a constant number of layers if the index is positioned at the end of the input. This holds true for low-parameter RNNs, SSMs, and masked linear-attention transformers. Conversely, small softmax transformers can accomplish this in a single layer, while non-masked linear-attention transformers require two layers, distinguishing them from their masked versions. When the index is at the start, small RNNs are capable of solving it.