Nautile-370M: Hybrid Reasoning Model with Spectral Memory

A new small language model named Nautile-370M has been unveiled, featuring 371 million parameters and designed for effective reasoning within limited parameter and inference constraints. This model utilizes a hybrid architecture that alternates between two SeqCond Attention (SCA) layers—drawing from the linear-time spectral sequence operator based on SeqCondenser—and one transformer layer. The objective is to merge the advantages of structured sequential models, such as long-context efficiency and state-tracking, with the flexible token-to-token routing provided by attention mechanisms. Training took place on a single Cloud TPU v4-64 pod slice through the Google TPU Research Cloud (TRC) program, followed by reinforcement learning on a single NVIDIA DGX Spark. The authors demonstrate that the SCA readout can precisely extract any token from the prefix summary and replicate softmax attention outputs, confirming SCA's expressiveness compared to standard attention.