SQLite Text-History Compression Prototype Yields 250x Reduction

Simon Willison, a well-known developer and blogger, has shared an in-depth exploration of an innovative method for storing revision histories in SQLite databases through compression. This concept, which originated during a walk with his dog, involves aggregating all prior versions of a text document into a JSON array and utilizing zlib or Zstandard (ZSTD) compression on the entire array. Willison engaged with GPT-Live voice mode in the ChatGPT iPhone app and later asked GPT-5.6 Sol Pro to create prototypes in Python. The prototype simulated 1,000 document revisions, compressing 20.4 MB of raw text to merely 80.3 KB with Zstandard, achieving a compression ratio exceeding 250:1. To minimize the need for decompressing and recompressing the entire array with each edit, the AI recommended dividing the history into several rows, each holding a maximum of 128 revisions or 3MB of uncompressed JSON. Dated 9th August 2026, the post features the complete transcript of the voice discussion and the generated files. Willison also offers a $10/month sponsorship option for a curated email summary of significant LLM advancements. This work is pertinent for developers and organizations handling extensive text archives, providing a viable approach for efficient versioning in relational databases.