L-PACT Framework Challenges Brain-Language Model Alignment Claims

A new study has introduced L-PACT, a framework designed to evaluate how well language models align with brain activity, arguing that just looking at prediction scores isn't enough. It examines various types of evidence, including predictive, relational, mechanism-stripping, and reliability-bounded evidence, using different neural datasets and model representations. The framework compares actual model features to nuisance baselines and rigorous controls, checks if model-to-brain patterns mirror brain-to-brain connections, and adjusts scores after mechanism stripping. The analysis includes 414 predictive-control rows, 2,304 relational profile rows, 4,320 mechanism-stripping rows, 420 brain-brain ceiling rows, and 146 integrated decisions. These results challenge the common belief that high prediction scores mean models are accurately reflecting brain-related language processing.