Federated Pre-Training Evaluation: Reliability of Downstream Fine-Tuning vs. Intrinsic Metrics

A recent study published on arXiv (paper 2607.28658) examines the effectiveness of evaluation methods for federated pre-training of foundation models. This approach allows for training on private or distributed datasets without centralizing them, but assessing performance is complicated by differences in client involvement and the availability of local data. The researchers highlight that the perplexity of pre-training tests relates closely to the distribution used for pre-training, while downstream benchmarks may require task-specific adjustments that do not necessarily indicate pre-training quality. They evaluate various protocols using a controlled group of centralized and federated-trained models, each with 16M parameters, trained on the same client data. The results are significant for those involved in federated learning and foundation model development, offering insights into which evaluation methods best reflect pre-training quality. The paper can be found on arXiv under the identifier 2607.28658.