Gender Bias in LLM Fake News Detection: First Systematic Study
A recent study published on arXiv (2608.03627) marks the first comprehensive examination of gender bias in fake news detection utilizing LLMs and real-world data. Researchers enhanced the LIAR benchmark by incorporating three gender variants of job titles (Neutral, Male, Female) for each statement to determine if truth assessments differ based solely on gender presentation. Six advanced LLMs were assessed using various bias and fairness metrics. All models demonstrated gender sensitivity, with 9.79%-35.13% of statements receiving differing labels among the three variants, and Male-Female comparisons revealing flip rates of 6.5%-23.6%. The study identified two main forms of bias: instability (inconsistent judgments) and directionality (systematic favoritism), emphasizing the importance of fairness in automated fact-checking systems.
Key facts
- First systematic investigation of gender bias in LLM-based fake news detection
- Uses real-world data from LIAR benchmark
- Three gender variants of speaker job titles: Neutral, Male, Female
- Six state-of-the-art LLMs evaluated
- 9.79%-35.13% of statements receive inconsistent labels across gender variants
- Male-Female comparisons show 6.5%-23.6% flip rates
- Two bias manifestations: instability and directionality
- Five models show statistically significant directional effects
Entities
Institutions
- arXiv