Role Prompts in LLMs Show Additive Structure but Resist Compression
A study on arXiv (2605.23147) investigates how instruction-tuned large language models process role prompts of the form "As X, do Y." Researchers found a clean linear decomposition at the prompt-to-answer transition in early/mid layers of Gemma-2-2B-IT and Qwen-2.5-{1.5B, 3B}-Instruct. Persona and task contribute through partially orthogonal additive directions, allowing substitution of a combined residual vector while preserving behavioral markers. However, contrary to expectations, the role prompt cannot be compressed into a single cached residual vector; even injecting the oracle clean residual fails.
Key facts
- arXiv paper 2605.23147
- Role prompts 'As X, do Y' show linear decomposition at prompt-to-answer transition
- Persona and task effects are additive and partially orthogonal
- Tested on Gemma-2-2B-IT and Qwen-2.5-{1.5B, 3B}-Instruct
- 12-cell short grid and 48-cell long-persona grid used
- Additive prediction cannot be cached into a single residual vector
- Oracle clean residual injection also fails
- Findings challenge compression assumptions
Entities
—