Deterministic Screen-Activity Compiler Enhances Agent Memory
A recent paper on arXiv (2608.05784) presents a deterministic, zero-model approach that transforms passively recorded screen interactions into organized 'activity frames' for computer-use agents. This innovative system divides local capture streams into categorized episodes, detailing application, website, timing, input volume, and evidence pointers, all without involving a model, thereby guaranteeing outputs that are byte-identical, cacheable, and subject to mechanical auditing. Evaluated on a professional's single-user dataset of 128,756 frames collected over 51 days, the compiler condenses a day's raw data into a context block that is 86 times smaller in just 68 milliseconds. An agent analyzing this block responds to daily inquiries with 98.4% accuracy (Wilson 95% CI 91.7-99.7%) compared to an independent oracle, surpassing LLM summaries, which yield 66-80% accuracy in a mid-tier mode. The authors contend that existing agent memory focuses on user speech rather than actions, resulting in unnecessary inference; this method effectively fills that void through deterministic compilation for enhanced memory efficiency and reliability.
Key facts
- Paper arXiv:2608.05784 introduces a deterministic, zero-model pipeline for compiling screen activity into agent memory.
- The pipeline segments local capture streams into typed activity frames with application, site, timing, input volume, and evidence pointers.
- No model is used in the loop, ensuring byte-identical, cacheable, and mechanically auditable outputs.
- Tested on a corpus of 128,756 frames over 51 active days from one professional user.
- The compiler reduces a day of raw capture to a prompt-ready context block 86x smaller in 68 ms.
- Agent reading the compiled block answers questions with 98.4% accuracy (Wilson 95% CI 91.7-99.7%).
- LLM summaries of the same capture achieve 66-80% accuracy in a mid-tier mode.
- The paper highlights that current agent memory records user statements, not actions, leading to redundant inference.
Entities
Institutions
- arXiv