G-Loss: Graph-Guided Fine-Tuning Improves Language Model Embeddings
A group of researchers has introduced G-Loss, a new graph-guided loss function aimed at enhancing models like BERT. Unlike traditional loss functions such as cross-entropy and contrastive methods, which operate mainly within local areas, G-Loss leverages semi-supervised label propagation to tap into the structural links found in the embedding manifold. It builds a document similarity graph that captures broader semantic connections, allowing the model to create more effective and resilient embeddings. When tested on five benchmark datasets—MR for sentiment analysis, R8 and R52 for topic categorization, Ohsumed for medical document classification, and 20NG for news categorization—G-Loss showed quicker convergence and produced more semantically consistent embedding spaces in many cases. This study is detailed in the arXiv preprint 2604.25853.
Key facts
- G-Loss is a graph-guided loss function for fine-tuning language models.
- It uses semi-supervised label propagation and a document-similarity graph.
- Captures global semantic structure, unlike local-only traditional losses.
- Evaluated on MR, R8, R52, Ohsumed, and 20NG datasets.
- Converges faster and produces more coherent embeddings.
- Targets downstream classification tasks: sentiment, topic, medical, news.
- Preprint available on arXiv (2604.25853).
- Applicable to models like BERT.
Entities
Institutions
- arXiv