Probabilistic Token Attribution for LLMs

A new research paper introduces a model-agnostic probabilistic token attribution measure for Large Language Models (LLMs). The method uses Bayes rule to invert next-token log-probabilities, capturing the model's internal representation of token sequence distributions independently of its computational structure. The attribution score is defined as the log ratio of the conditional probability of the response given the prompt versus the probability with a token marginalized away. This approach situates LLMs within stochastic process theory, offering a framework to understand how models generate responses based on learned probabilities. The paper is available on arXiv under identifier 2605.21726.