Intern-S2-Mobius: Decoupling Knowledge and Reasoning in Foundation Models
A new architecture for foundation models, named Mobius-v0, has been unveiled by researchers. This innovative design distinguishes between knowledge storage and reasoning functions. It features a globally accessible Memory (FFN) for knowledge vector storage, alongside multiple Reasoners (Self-Attn) that perform iterative compositional reasoning. Reasoners utilize hidden states as both cache and carriers, querying memory for necessary knowledge vectors, which are then relayed back to the reasoning operators. This separation enhances both knowledge compression and reasoning efficiency. A 7B model developed from scratch based on Mobius-v0 matches the downstream performance of a 7B Transformer baseline while utilizing only 62.6% of its training data. Furthermore, Intern-S2-Mobius, which is continually pretrained from Qwen3.5-35B, achieves comparable downstream results with nearly 4x faster end-to-end inference. The research is documented on arXiv in the Computer Science > Artificial Intelligence section, contributing to advancements in AI and machine learning.
Key facts
- Mobius-v0 architecture separates knowledge storage (FFN) from reasoning (Self-Attn).
- Globally shared Memory stores knowledge vectors; multiple Reasoners perform compositional reasoning.
- Hidden states serve as cache and carrier for knowledge queries and transmission.
- 7B model trained from scratch achieves similar downstream scores as 7B Transformer baseline with 62.6% of training data.
- Intern-S2-Mobius, continually pretrained from Qwen3.5-35B, achieves similar downstream scores with nearly 4x end-to-end inference speedup.
- Paper submitted to arXiv under Computer Science > Artificial Intelligence.
- arXiv ID: 2608.14290.
- Research focuses on improving knowledge compression and reasoning efficiency in foundation models.
Entities
Institutions
- arXiv