New Rule-Generation Method for LLM Compositionality Estimation
A novel method for assessing compositionality in large language models (LLMs) has been introduced, tackling the shortcomings of conventional compositional generalization assessments. This technique, detailed in an arXiv paper (2604.27340), involves LLMs creating a program that serves as rules for mapping datasets, subsequently evaluating compositionality through a complexity-based theory. This approach effectively addresses two significant challenges: the lack of clarity in output-only evaluations and the issue of combination leakage from dataset splits. Experiments were carried out on advanced LLMs utilizing string-based tasks. This research offers a fresh analytical framework for comprehending LLM compositionality.
Key facts
- arXiv paper ID: 2604.27340
- Proposes rule-generation perspective for compositionality estimation
- Addresses explainability defects of compositional generalization tests
- Solves combination leakage issues from dataset partition
- Uses complexity-based theory for estimation
- Requires LLMs to generate program rules for dataset mapping
- Experiments conducted on existing advanced LLMs
- Focuses on string-based tasks
Entities
Institutions
- arXiv