Tabular Foundation Models Degrade Under Distribution Shifts
A new empirical study evaluates the out-of-distribution (OOD) performance of nine Tabular Foundation Models (TFMs), including TabPFNv2, TabPFNv2.5, TabPFNv2.6, TabPFNv3, TabICL, TabICLv2, Mitra, LimiX, and TabFM. These models, which have shown competitive predictive performance on independent and identically distributed data, are tested on three real-world datasets from the TableShift study: HELOC, Voting, and Childhood Lead. These datasets cover label, socioeconomic, and geographic shift types. The results reveal that all evaluated TFMs degrade systematically under distribution shift, highlighting a critical limitation for real-world applications where data distributions often change.
Key facts
- Nine Tabular Foundation Models were evaluated: TabPFNv2, TabPFNv2.5, TabPFNv2.6, TabPFNv3, TabICL, TabICLv2, Mitra, LimiX, and TabFM.
- Three datasets from the TableShift study were used: HELOC, Voting, and Childhood Lead.
- The datasets cover label, socioeconomic, and geographic shift types.
- All TFMs showed systematic degradation under distribution shift.
- The study is empirical and focuses on out-of-distribution performance.
- TFMs are novel approaches for tabular predictive tasks.
- Most TFMs are trained and evaluated on i.i.d. data, which is unrealistic for real-world scenarios.
- Limited research has been conducted on TFMs under distribution shifts.
Entities
Institutions
- TableShift