Tensor Memory: Fixed-Size Recurrent State for Long-Horizon Transformers
A recent study published on arXiv introduces Tensor Memory, a component designed to enhance Transformer blocks by incorporating a fixed-size 3D memory tensor that operates recurrently. This innovation tackles the challenge of memory expansion in traditional Transformers, especially relevant for tasks like long-horizon video comprehension and reasoning that is sensitive to occlusions. The module employs a differentiable soft write mechanism to insert data into a voxel grid, utilizes efficient local interaction operators, and implements gated recurrent dynamics. Tensor Memory effectively separates state capacity from the input length while maintaining a spatial inductive bias.
Key facts
- Tensor Memory is a lightweight module for Transformers.
- It uses a fixed-size recurrent 3D memory tensor.
- Tokens write into a voxel grid via differentiable soft write.
- Memory is updated with local interaction and gated recurrent dynamics.
- Tokens read back context via continuous sampling with gated residual fusion.
- Tensor Memory decouples state capacity from input length.
- It preserves a spatial inductive bias.
- The paper is on arXiv with ID 2605.27686.
Entities
Institutions
- arXiv