LLMs Show Self-Preference Bias in Assessing Idea Originality

A recent investigation available on arXiv examines the alignment of Large Language Models (LLMs) with human evaluators in judging the originality of responses during a divergent thinking exercise. The study involved an analysis of 4,813 submissions to the Alternate Uses Task (AUT) from both highly creative and less creative individuals, as well as ChatGPT-4o. Two university students, who received extensive training, served as human raters. The machine evaluation was conducted using two tailored systems, OCSAI and CLAUS, along with ChatGPT-4o, which followed the same instructions as the human raters. The results indicate a preliminary self-preference bias in automatic systems, favoring responses aligned with their own style, underscoring the necessity for careful calibration in employing LLMs for assessing creativity.