AgentMemBench: Benchmarking Long-Term Memory Strategies for Conversational AI
A new evaluation framework named AgentMemBench has been launched to assess the capabilities of conversational AI in maintaining long-term memory during interactions. This benchmark tackles the limitations of context windows affecting memory retrieval in extended conversations. It evaluates five memory strategies: in-context windowing, external key-value stores, graph-based episodic memory, compression-based summarization, and web-augmented memory. The assessment utilizes three public datasets for various dialogue types. Initial findings suggest that the external key-value store method outperforms the others, although comprehensive results are pending. The entire framework is accessible via arXiv under the identifier 2608.00009.
Key facts
- AgentMemBench is a unified, reproducible benchmark for evaluating long-term memory management in conversational AI.
- It compares five memory strategies: ICW, EKV, GEM, CBS, and WAM.
- Evaluation uses three public datasets: LoCoMo, MultiDoc2Dial, and MSC.
- Metrics include Recall@k, MRR, nDCG@k, Answer F1, Faithfulness, Memory Footprint, and Latency.
- 491 annotated question turns are used for evaluation.
- Qwen2.5-7B-Instruct (4-bit) is used for generation and judging with greedy decoding.
- EKV dominates on every qualitative metric according to preliminary results.
- The benchmark is published on arXiv under identifier 2608.00009.
Entities
Institutions
- arXiv