VeriTaS: Dynamic Benchmark for Multimodal Fact-Checking
A team of researchers has launched VeriTaS (Verified Theses and Statements), marking the debut of a dynamic benchmark for multimodal automated fact-checking (AFC). This new benchmark overcomes the shortcomings of current static benchmarks, which are prone to data leakage when their claims are incorporated into LLM pretraining datasets. VeriTaS includes 25,000 authentic claims sourced from 104 professional fact-checking organizations in 54 different languages, encompassing both text and audiovisual material. Every quarter, claims are integrated through a fully automated seven-stage process that standardizes formats. This benchmark is crafted to withstand extensive pretraining of foundational models, ensuring that performance metrics accurately reflect true verification capabilities.
Key facts
- VeriTaS is the first dynamic benchmark for multimodal automated fact-checking.
- It includes 25,000 real-world claims from 104 fact-checking organizations.
- Claims cover 54 languages and both textual and audiovisual content.
- New claims are added quarterly via an automated seven-stage pipeline.
- The benchmark addresses data leakage in static benchmarks used for LLMs.
- It is designed to remain robust under ongoing large-scale pretraining.
- The work is published on arXiv under identifier 2601.08611.
- The goal is to reliably reflect actual claim verification abilities.
Entities
—