Conditional Attribute Transformers for Sequence-Level Control

arXiv preprint 2605.14004 introduces Conditional Attribute Transformers, a method that extends autoregressive generative models to estimate sequence-level attributes without modifying the input or requiring expensive sampling. The approach jointly predicts the next token and the attribute value conditional on each token choice, enabling per-token credit assignment and counterfactual analysis in a single forward pass. This addresses limitations of standard next-token prediction, which overfits local patterns and underfits global structure. The method allows identifying which tokens in a sequence are associated with an attribute's value and performing counterfactual analysis. The work is relevant for applications needing attribute estimation or control in generated sequences.