Predicting Compaction Utility from Lakehouse Table Metadata

An arXiv paper (2608.08639) presents a novel open simulation framework designed to assess the effectiveness of data compaction in open lakehouse table formats. This framework produces 2,376 Apache Iceberg tables across three file size scales, extracts 17 metadata characteristics from manifest files without accessing the data, and employs an XGBoost model to forecast the continuous file-reduction ratio, achieving an R² of 0.998 and RMSE of 0.013. Interestingly, the binary decision for compaction is easily separable by a single threshold at the partition level (max_files_per_partition > 4), eliminating the need for a learned model. Validation across 96 TPC-H tables demonstrates generalization without retraining (R² = 0.976). A query benchmark indicates that while compaction aids metadata-heavy queries, it may hinder full-scan aggregations by decreasing task parallelism. The study tackles the challenge of determining the optimal timing for compaction, which is presently driven by thresholds, and seeks to identify which metadata features influence compaction utility. The framework is open for further exploration and practical implementation to enhance lakehouse performance.