Protocol Flaws Inflate AutoML Benchmark Results at Short Budgets
A study available on arXiv (2608.07303) indicates that the AutoML engine Orcetra outperformed both FLAML and AutoGluon across 513 OpenML datasets, achieving a success rate of 57.1% within a 60-second timeframe and 78.4% when compared solely to FLAML at 30 seconds. However, these results were influenced by flaws in the protocol: the search loop evaluated all candidates based on the test split, reporting the highest score, which skewed the results due to numerous noisy estimates. In contrast, the baselines relied on training data, interacting with the test set only once. Furthermore, although the budget was assessed prior to candidate execution, it was not enforced during, resulting in Orcetra averaging 120 seconds, 2.24 times longer than AutoGluon. The authors intend to refine their methodology by shifting selection to a validation set in future experiments.
Key facts
- Orcetra appeared to beat FLAML and AutoGluon on 513 OpenML datasets.
- Orcetra won 57.1% of datasets at a nominal 60-second budget.
- Orcetra won 78.4% of datasets against FLAML alone at 30 seconds.
- The search loop scored every candidate on the test split and reported the best.
- Baselines selected on training data and touched the test set once.
- The budget was checked before launching a candidate but never enforced during one.
- Orcetra consumed a median of 120 seconds against a 60-second budget.
- Orcetra used 2.24 times the wall-clock time of AutoGluon.
Entities
Institutions
- arXiv
- OpenML