Transformer Models Show Dual Geometry in Concept Representation

A recent study featured on arXiv investigates the role of causal inner product derived from unembedding covariance in enhancing concept transfer between languages via transformer architectures. The research analyzed 17 models across four different language pairs and found that Whitened Causal Alignment performs similarly to spectral regularization, achieving a statistical significance level of p = 0.95. Notably, differences among five model types displayed a significant anti-concentration (p < 10^{-33}), backed by SAE feature analysis (p = 4.5 × 10^{-19}). Key observations include unique geometric patterns in activation spaces and effective interventions on models Gemma and Llama, marking Cohen's d values of up to 1.8.