BCMT: A New Transformer Architecture for Efficient Long-Context Language Modeling

A new architecture called BCMT (Blockwise Causal Memory Transformer) has been developed by researchers for modeling long-context language. This innovative design tackles the quadratic complexity associated with dense self-attention found in traditional Transformers by separating local token interactions from the propagation of global context. In BCMT, local blocks utilize dense causal self-attention independently, and each block produces an adaptive summary that is combined using an exponential causal memory. This memory is reintegrated into token representations, facilitating the efficient transfer of long-range contextual information without the need for global attention. Unlike conventional Transformers and recurrent memory systems, BCMT eliminates dense interactions among distant tokens and does not depend on learned memory states. Its memory system is fully parallelizable and compatible with standard training and inference methods. The paper can be accessed on arXiv with the identifier 2608.13578, categorized as 'cross'. This advancement may greatly enhance the efficiency of processing lengthy sequences in natural language processing applications.