Data Quality Over Capacity: LoRA Adapters for Closed-Book QA

A recent arXiv study (2607.21861) explores the integration of documents into the weights of a 4-bit Gemma-4-e4b model using LoRA for closed-book question answering, thereby bypassing retrieval and context-window limitations. Researchers conducted around 100 training runs, ranging from single documents to a 99-document corpus. They discovered that once the adapter's capacity is sufficient, the quality of the training data becomes the primary factor influencing closed-book accuracy, surpassing the effects of LoRA rank, learning rate, and two other combined architectures. A single curation pass—condensing gold answers to canonical 1-6 word spans and removing trivia—boosted accuracy from 57.7% to 85.7% on a 15-document corpus, outperforming any architectural modification. The study also identifies a trend where rank must increase with corpus size, revealing a previously misinterpreted relationship between rank and learning rate. A real retrieval-augmented generation (RAG) baseline was included for comparison on a 15-document subset.