ProtDBench: Standardized Benchmark for Protein Binder Design

ProtDBench introduces a novel framework for evaluating de novo protein binder design, focusing on standardized and throughput-aware assessments to remedy the inconsistency in evaluation methods across various studies. It establishes common benchmark tasks, evaluation protocols, and criteria for success, facilitating a systematic examination of how evaluation design influences performance outcomes. By utilizing a comprehensive wet-lab annotated dataset, the framework scrutinizes widely used structure prediction models as evaluators, uncovering significant verifier-dependent biases and low agreement under the same filtering criteria. Additionally, it benchmarks a selection of open-source generative binder design techniques across ten distinct protein targets using a consistent evaluation protocol. ProtDBench also includes throughput-aware metrics that extend beyond simple per-sequence success rates.