Test-Driven Data Engineering for Self-Improving LLMs from Raw Corpora

A recent paper on arXiv (2604.24819) introduces a framework that aligns the data-engineering lifecycle for large language models (LLMs) with the software development lifecycle. The authors contend that the process of fine-tuning using domain-specific corpora lacks adequate feedback mechanisms to identify deficiencies in training data. They propose leveraging a structured knowledge representation derived from the source corpus as a common basis for both training and evaluation data. In this model, training data acts as source code detailing the learning objectives for the model, while training equates to compilation, benchmarking to unit testing, and data repair driven by failures to debugging. This methodology aims to facilitate self-improving LLMs by offering a means to identify and rectify data issues when models struggle with domain-specific tasks.