Steerling-8B: Interpretability as a Training Constraint Scales with Capability

A recent research paper disputes the traditional belief that there is a trade-off between interpretability and model performance. The authors advocate for treating interpretability as a constraint integrated into the training process, optimized alongside the language modeling goal. Their findings, based on three orders of magnitude in computational power across both autoregressive and diffusion language models, reveal that as models increase in size, interpretability improves, leading to representations that are more disentangled and aligned with human concepts. They demonstrate this method using Steerling-8B, a diffusion language model featuring a causal attention mask, which can link generated tokens to pertinent input tokens, comprehensible concepts, and training data. This paper can be found on arXiv with the identifier 2608.07594v1 and is categorized as a cross announcement. This research indicates a potential shift in integrating interpretability into AI development, which may affect future model design and transparency standards.