Billion-Parameter Language Models Achieve Exact State Deletion

A recent paper on arXiv (2508.12220) reveals that deletion of state-exact, trace-preserving information is feasible in billion-parameter language models. This study introduces a counterfactual framework that allows for the precise reproduction of a deletion counterfactual after certain examples are removed from the replay dataset. The approach requires prospective instrumentation, necessitating that the original training run captures execution provenance and maintains an uncontaminated checkpoint. In single-GPU settings, replaying from a token store without the specified rows reconstructs a trace oracle bit-for-bit in both model and optimizer states. The findings confirm exactness for Pythia 160M across four deletion geometries, while Pythia 2.8B shows all 2,775,208,960 model-state elements matching for a random 5% deletion. Llama 3.2 1B also achieves exactness after excluding 400 of 4,000 TOFU examples. Although these results affirm billion-parameter state exactness, the authors caution that the computational cost of deletion remains significant. This paper marks a notable advancement in machine unlearning, with potential consequences for privacy, compliance, and model editing.