LeakGauge: New Method Measures Context-Leakage Signals in Large Language Models
A preprint has been uploaded to arXiv (identifier 2608.17829v1) by researchers introducing LeakGauge, a technique designed to identify signs of external context leakage in large language models (LLMs). These models frequently depend on external context, which can lead to vulnerabilities where malicious inputs might expose sensitive data. LeakGauge assesses leakage prior to decoding by adding a suffix to the query, allowing for the evaluation of leakage behavior and linking prefill token probabilities to an attack-risk score. Tests conducted on 11 LLMs, such as the 753B-parameter GLM-5.2 and the 2.8T-parameter Kimi-K3, indicate that a content-agnostic gauge provides more reliable signals. This method could improve LLM security without needing access to hidden representations, highlighting the significance of choosing the appropriate gauge signal type.
Key facts
- The paper is available on arXiv with identifier 2608.17829v1.
- LeakGauge is a method for measuring context-leakage signals in large language models.
- LLMs rely on external contexts like system prompts and retrieved documents, creating an attack surface.
- Adversarial inputs can induce models to disclose external context.
- Prior probing studies required hidden states, posing deployment challenges.
- LeakGauge appends a suffix to the query and maps prefill token probabilities to an attack-risk score.
- A content-agnostic gauge verbalizing leakage behavior is more robust than a direct gauge using initial tokens of confidential content.
- The method was tested across 11 LLMs, including GLM-5.2 (753B) and Kimi-K3 (2.8T).
Entities
—