Crowdsourced Audiovisual Deepfake Detection Shows Human Limitations
A recent investigation published on arXiv (2605.04797) explores how crowd workers identify audiovisual deepfakes. The researchers conducted two parallel crowdsourcing studies on Prolific, utilizing the AV-Deepfake1M and Trusted Media Challenge (TMC) datasets, which included 48 videos from each dataset (a total of 96) and gathered 960 assessments (10 for each video). Findings indicate that crowd workers seldom misidentify genuine videos as altered, yet they overlook numerous manipulations, exhibiting limited consensus across the videos. While aggregating multiple evaluations per video enhances the authenticity signal, it does not recover the overlooked manipulations. This study underscores the dependability of human evaluations in combating misinformation as deepfakes grow more convincing and simpler to create.
Key facts
- Study published on arXiv with identifier 2605.04797
- Focuses on audiovisual deepfake detection by crowd workers
- Uses AV-Deepfake1M and Trusted Media Challenge (TMC) datasets
- Two matched crowdsourcing studies conducted on Prolific
- 48 videos sampled per dataset, 96 total
- 960 judgments collected, 10 per video
- Crowd workers rarely misclassify authentic videos as manipulated
- Many manipulations are missed by crowd workers
- Agreement among workers remains limited across videos
- Aggregating multiple judgments stabilizes authenticity signal but cannot recover missed manipulations
Entities
Institutions
- arXiv
- Prolific
- AV-Deepfake1M
- Trusted Media Challenge (TMC)