Gradient-Based Explanations of LLMs Align with Brain Activity

A recent study published on arXiv (2502.14671) explores the connection between Large Language Model (LLM) representations and brain activity during language processing. Researchers aimed to determine if explainable AI (XAI) could shed light on this connection by employing attribution methods to assess the influence of each input word on the LLM's next-word predictions. They subsequently utilized these insights to forecast fMRI data from participants engaged in narrative listening. The results indicated that gradient-based attribution methods align closely with brain activity, providing unique variance beyond acoustic and word-rate factors, and surpassing internal representations in early auditory areas. By applying conductance, the study revealed that early layers exhibit heightened sensitivity to word types and preferential alignment with auditory regions, while later layers correspond with higher-order language areas. These findings imply that gradient-based explanations illuminate aspects of neural language processing overlooked by internal representations, presenting a novel approach to understanding the brain-language connection.