LLMs Struggle with Deception in Secret Hitler Game

A recent study published on arXiv presents a new open-source framework aimed at assessing the ability of Large Language Models (LLMs) to mislead players in the board game Secret Hitler. Researchers developed novel metrics like Role Identification Accuracy and Deception Retention Rate to measure LLM performance. They found that LLMs lagged behind both human players and rule-based systems in conversational effectiveness compared to strategic capabilities. Despite using Chain-of-Thought prompting, LLMs struggled, with fascist characters winning 23.2% less frequently. In contrast, rule-based agents achieved 86.7% alignment with expert human voting, revealing significant challenges in LLM deception evaluation.