Poli-Bias: New Framework Measures Political Bias in LLMs Across International Conflicts
A new tool named Poli-Bias has been developed to evaluate political biases in large language models (LLMs), particularly in the context of international conflicts. This innovative method addresses the nuanced challenges in identifying biases that arise from subtle variations in information presentation. Poli-Bias analyzes how different country identities influence responses to various geopolitical prompts and legal discussions, breaking bias down into five distinct categories for better clarity. A study reviewing 13 contemporary LLMs demonstrated that user backgrounds and national contexts impact the treatment of similar actions. For more details, the research can be found on arXiv, reference number 2608.06123.
Key facts
- Poli-Bias is a counterfactual framework for measuring political bias in LLMs.
- It compares responses to paired prompts with swapped country identities.
- The framework decomposes response disparities into five interpretable dimensions.
- 13 contemporary LLMs were evaluated.
- Country identities and user affiliations can systematically affect LLM responses.
- The paper is available on arXiv with identifier 2608.06123.
- The research focuses on international political conflicts.
- The framework addresses bias in framing, argumentation, and legal reasoning.
Entities
Institutions
- arXiv