Clinician-Authored Rubrics for Evaluating Clinical AI Systems
A recent study presents a specialized rubric methodology designed to assess clinical AI documentation systems, focusing on achieving a balance among clinical validity, economic feasibility, and responsiveness to ongoing changes. In total, 1,646 rubrics were created by twenty clinicians for 823 clinical cases, which included 736 real-world and 87 synthetic cases, spanning primary care, psychiatry, oncology, and behavioral health. Validation involved confirming that an LLM-based scoring agent consistently rated outputs favored by clinicians higher than those that were rejected. The research evaluated seven iterations of an EHR-integrated AI agent, investigating if LLM-generated rubrics could reflect clinician consensus, thereby addressing the challenges of costly and time-consuming expert evaluations. Findings indicated that clinician-authored rubrics successfully distinguished between high- and low-quality outputs, facilitating safer, iterative deployment of clinical AI by lessening the need for manual expert assessments.
Key facts
- Twenty clinicians authored 1,646 rubrics for 823 clinical cases
- Cases included 736 real-world and 87 synthetic encounters
- Covered primary care, psychiatry, oncology, and behavioral health
- Rubrics validated by confirming LLM scoring agent preferred clinician outputs
- Seven versions of an EHR-embedded AI agent were evaluated
- Method aims to be clinically valid, economically viable, and sensitive to iterative changes
- Study examines if LLM-generated rubrics can match clinician agreement
- Clinician-authored rubrics effectively discriminated between high and low quality outputs
Entities
—