Scale-Aware Memorization Diagnostics for Code LLMs

A new arXiv preprint (2608.12771) investigates the reliability of memorization detection techniques in large language models for code, revealing that traditional probing methods fail at scale. The study finds that encoder-style probes using perturbations like synonym fuzzing or dead-code insertion, and decoder-style probes relying on log probabilities, degrade severely when applied to larger models, even on benchmarks known to be contaminated. The authors argue that memorization and representation load should be treated as separate phenomena, and they propose using invertible mathematical transforms on numeric problems to isolate these factors. The research highlights the need for scale-aware diagnostics in evaluating code LLMs, as current methods may underestimate memorization in scaled models. The preprint was announced on arXiv with a cross-type classification, and the full text is available at the provided URL.