Layer Equivalence Tests Diverge: Replacement vs. Interchange in Transformers

A recent study published on arXiv (2605.16234) indicates that two prevalent techniques for assessing layer equivalence in transformer neural networks—replacement and interchange—can produce notably different outcomes. The replacement method evaluates if one layer's mapping can effectively replace another in its original context, whereas the interchange method checks if two layers can be swapped without significant impact. The findings reveal that for pretrained transformers, the disparity in protocols can alter the number of layers deemed safe for pruning by several times under identical evaluators, particularly with high replacement distances. Observations across various checkpoints and architectures, including Pythia 410M and 1.4B, demonstrate that this gap increases from initialization to convergence. At the 8B scale, Qwen3-8B enters a divergent state where interchange-guided removal is significantly safer than replacement-guided methods at equivalent layer budgets, while Llama-3.1-8B shows more consistency between the two approaches.