New Metrics C-FEX and PKP Evaluate RAG Fine-Tuning for Technical Document Generation
A recent pre-print available on arXiv (identifier 2603.23047v2) explores an area in retrieval-augmented generation (RAG) fine-tuning, specifically targeting document generation within the field of electronics engineering. The research modifies two language models, each with 7 billion parameters, for supervised RAG fine-tuning (RAG-SFT) aimed at producing requirements documents while analyzing various data strategies. The authors present C-FEX, a claim-based evaluation framework, alongside Parametric Knowledge Precision (PKP) to evaluate claims derived from the model's memory against the retrieved context. They reformulate an existing metric into a product of PKP and parametric rate (PR). This study improves RAG evaluation for industrial use, providing valuable insights on data selection for fine-tuning and metrics for assessing factual accuracy in technical fields.
Key facts
- Research paper identifies arXiv ID 2603.23047v2.
- Study applies RAG-SFT to generate requirements documents.
- Domain is electronics engineering.
- Two 7B language models are adapted.
- Two training data strategies are compared.
- C-FEX is a claim-based evaluation pipeline that attributes claims to their origin.
- PKP isolates and evaluates claims from the model's weights.
- Prior metric for parametric knowledge decomposes as PKP times PR.
Entities
—