New Benchmark Design Approach for Generative AI in Journalism
A research paper from arXiv (2511.05501) proposes a human-centered design process for creating generative AI benchmarks with improved real-world validity, specifically for journalism. The study involved 23 journalism professionals in a workshop to inform a domain-oriented evaluation "cookbook." Findings highlight challenges in translating tasks to evaluation constructs, aligning metrics with domain values, and balancing stakeholder needs. The work addresses criticisms that existing benchmarks lack ecological and construct validity.
Key facts
- arXiv paper 2511.05501 proposes human-centered benchmark design for generative AI
- Study focused on journalism domain with 23 professionals
- Workshop informed a domain-oriented evaluation cookbook
- Findings include challenges in task-to-construct translation
- Challenges in aligning metrics with domain-specific values
- Challenges in balancing stakeholder needs
- Existing benchmarks criticized for lack of ecological validity
- Existing benchmarks criticized for lack of construct validity
Entities
Institutions
- arXiv