MemArena: New Benchmark for On-Device Personal Memory Assistants

A new benchmark called MemArena has been introduced to evaluate on-device personal memory assistants that handle private interpersonal conversations using open-weight models. The benchmark, detailed in a paper on arXiv (2608.02613), addresses gaps in existing memory benchmarks by combining activity-dense interaction, ego-centric perspective, and coherent multi-session worlds. MemArena is built with the MASim agent simulator, simulating 50 agents over 15 days, generating 10.3 million dialog-text tokens and 24.1K text-only ego-observed tokens per agent per day. It co-generates ground truth across six evaluation dimensions: recall, reasoning, and trustworthiness. The study evaluates five open-weight readers with different memory backends: Vanilla context, BM25-RAG, Oracle retrieval, Memobase, and MemSearch. Key findings show that the choice of memory backend significantly impacts content accuracy. For instance, at Qwen3-0.6B, switching from Memobase to MemSearch yields gains of +32.5 and +19.2 percentage points, which exceeds the gains from scaling the reader model (+10.6 and +6.8 percentage points). This suggests that optimizing memory infrastructure may be more critical than increasing model size for these tasks. The benchmark aims to advance the development of edge-deployed personal memory assistants, emphasizing privacy and on-device processing.