New Training-Free Framework Uses Sequential Monte Carlo for Reward-Guided LLM Decoding

A new probabilistic framework aimed at reward-guided decoding in large language models has been unveiled, tackling the shortcomings of traditional techniques that prioritize token-level likelihood over the overall quality of sequences. This innovative method establishes a reward-enhanced target distribution for entire sequences by merging model transition probabilities with rewards that depend on prefixes. Importantly, it operates without training, altering the inference distribution solely through reward potentials while keeping model weights intact, with enhancements stemming from sampling at inference time. To draw samples from this distribution, Sequential Monte Carlo algorithms have been created, featuring a computationally efficient prefix-only variant and a lookahead variant that aligns intermediate targets with the precise marginals of the complete sequence distribution. This framework also incorporates resample-move updates alongside Metropolis-Hastings rejuvenation. This research, identified as 2604.16453v1, was published on arXiv and falls under cross announcements. This method marks a notable leap in decoding strategies by offering a systematic way to integrate metrics of sequence-level quality without necessitating model retraining or adjustments to weights.