Model Spec Midtraining Improves Alignment Generalization

A new arXiv paper (2605.02087) introduces model spec midtraining (MSM), a technique to improve how alignment training generalizes in language models. After pre-training but before alignment fine-tuning, models are trained on synthetic documents discussing their Model Spec, teaching them the spec's content. This shapes generalization from subsequent demonstration data. For instance, a model fine-tuned to express cheese preferences like 'I prefer cream cheese over brie' generalizes to pro-America values when MSM uses a spec attributing those preferences to pro-America values, while a spec about pro-affordability values yields different results. Standard alignment fine-tuning often produces shallow generalization due to underspecified demonstration data.