Caracal: Efficient LLM Architecture Using Fourier Transform

Caracal is a novel architecture for Large Language Models that replaces the quadratic-cost attention mechanism with a parameter-efficient Multi-Head Fourier (MHF) module, achieving O(L log L) complexity. It uses Fast Fourier Transform (FFT) for sequence mixing and introduces a frequency-domain causal masking technique via asymmetric padding and truncation to enable autoregressive generation. Unlike hardware-specific models like Mamba, Caracal relies on standard library operators, ensuring portability. Evaluations show competitive performance with existing models. The paper is available on arXiv.