s-Trace Method Reveals Two-Phase Computation in LLMs

A new method called s-Trace efficiently estimates the minimal subgraph of a transformer-based large language model (LLM) that best approximates the full model output. Applying s-Trace to various LLMs reveals that computation is organized in two distinct phases: an early phase where a small subgraph, mostly from early layers, reconstructs the head of the output distribution, and a later phase where additional nodes, primarily attention heads in later layers, provide incremental refinements. The amount of computation needed per input correlates with model uncertainty, and sparser subgraphs encode shallow statistics like unigram frequency. The findings suggest that LLMs do not exploit their full capacity for all inputs.