EmbGen: Synthetic Data Pipeline for Domain-Specific LLM Training

EmbGen is a pipeline for generating synthetic data that tailors small instruction-tuned models to specific domains. It breaks down a domain corpus into pairs of entity descriptions, which are then reconstructed using semantic structures based on embedding similarities. Question-answer pairs are produced through proximity sampling, as well as intra-cluster and inter-cluster sampling with prompts tailored to specific clusters. The performance of EmbGen was assessed against EntiGraph, InstructLab, and Knowledge-Instruct across three datasets exhibiting diverse semantic heterogeneity, utilizing fixed token budgets of 5 and 20 million tokens. This approach seeks to lower the expenses associated with gathering curated instruction-response examples for supervised fine-tuning.