ARC-Encoder: Compressing Text for LLMs
A new encoder called ARC-Encoder compresses text into continuous representations that replace token embeddings in decoder LLMs, reducing inference costs without fine-tuning or modifying the target model. The systematic study of training strategies and architecture choices led to its design, outputting x-times fewer representations (typically x∈{4,8}) than text tokens. Evaluated across various LLM usage scenarios, including in-context learning, ARC-Encoder offers an alternative to context compression techniques that degrade general abilities when used for specific purposes. The approach addresses longer contexts and increased inference costs from techniques like retrieval-augmented generation or chain-of-thought reasoning.
Key facts
- ARC-Encoder compresses context into continuous representations replacing token embeddings.
- It outputs x-times fewer representations, typically x∈{4,8}.
- No fine-tuning or architecture modification of the target LLM is required.
- Systematic study of training strategies and architecture choices informed design.
- Evaluated across various LLM usage scenarios including in-context learning.
- Addresses increased inference costs from longer contexts.
- Alternative to methods that degrade general abilities.
- Related to retrieval-augmented generation and chain-of-thought reasoning.
Entities
—