Context Sampling Strategies for TabPFN on Small Tabular Datasets
A study on arXiv (2607.26628) investigates how context selection affects TabPFN, a method for classification via in-context learning on small tabular datasets. Using 15 OpenML datasets, researchers examined three questions: whether larger contexts reduce prediction variability, whether accuracy depends on preserving training distribution or feature-space coverage, and whether expensive selection methods (K-Means, farthest-point sampling) outperform uniform random sampling. Results show larger contexts improve both accuracy and stability, with AUC coefficient of variation dropping from 6–18% at k=16 to lower values. Expensive selection methods offered no benefit over random sampling. The findings guide practitioners in choosing context size and sampling method for TabPFN.
Key facts
- TabPFN uses in-context learning for classification without gradient updates.
- Study conducted on 15 OpenML datasets.
- Larger contexts reduce prediction variability and improve accuracy.
- AUC coefficient of variation decreased from 6–18% at k=16.
- K-Means and farthest-point sampling provided no advantage over uniform random sampling.
- Paper available on arXiv with ID 2607.26628.
- Research focuses on small tabular datasets.
- Context size and row selection are critical choices for practitioners.
Entities
Institutions
- arXiv
- OpenML