Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese

Researchers have unveiled Prosa, the inaugural multi-turn chat benchmark for Brazilian Portuguese users, featuring 1,000 conversations from WildChat. The findings indicate that employing binary rubric scoring with multi-judge filtering effectively removes biases associated with judge models, a limitation seen in holistic scoring. With filtered rubric scoring, consensus among three judges from diverse model families was achieved for all 16 model ranks, in contrast to only 7 ranks under holistic scoring. Additionally, the filtering pipeline boosts the average score difference between adjacent models by 47%, improving discriminative capabilities. Evaluating a new model using Prosa costs around $2.1 with Gemini 3 Flash as the evaluator. The benchmark and filtering code have been made publicly available.