Neural Representations Require Symmetry-Stable Probes

A recent study published on arXiv (2605.11448) contends that neural representations are not singular entities; various systems executing the same computation may possess hidden coordinates that differ due to reparameterization. The researchers suggest that probe families designed to uncover representation structures should remain stable under relevant symmetries instead of being linked to a specific basis. Their investigation focuses on the final readout layer, where equivalent realizations lead to affine transformations. This symmetry principle establishes a distinct hierarchy of shallow coordinate-stable probes, with linear probes representing the first degree. Additionally, they argue that cross-model probe transfer should utilize a shared probe-visible quotient, reflecting the representation modulo directions that the probe family cannot detect, rather than the complete hidden state. Experiments conducted on both synthetic and real-world tasks validate these findings.