AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse
AdapShot is a new method for many-shot in-context learning (ICL) that dynamically determines the optimal number of examples per query, addressing the limitations of static shot counts. It uses a probe-based evaluation mechanism with output entropy to adapt shot counts, and incorporates semantics-aware KV cache reuse to reduce computational and memory costs. The approach aims to improve reasoning performance while enabling efficient inference for large language models.
Key facts
- AdapShot dynamically optimizes shot counts for each query.
- It uses output entropy to determine the optimal number of shots.
- Semantics-aware KV cache reuse reduces prefilling computation.
- The method addresses limitations of static shot counts in many-shot ICL.
- It aims to improve reasoning performance of LLMs.
- The approach reduces computational and memory costs of long contexts.
- AdapShot is proposed as a solution for efficient many-shot ICL.
- The method is described in arXiv paper 2605.03644.
Entities
Institutions
- arXiv