ClinPivot Benchmark Tests AI Clinical Decision-Making
ClinPivot has been launched by researchers as an auditable standard to assess if clinical foundation models modify treatment choices when patient circumstances vary. This benchmark employs biomedical relationships and altered patient contexts to determine if models adapt their decisions in response to new clinical parameters. Results indicate that high performance in medical QA does not consistently forecast decision-making accuracy; leading models and task-specific Qwen adaptations frequently struggle with proper pivoting, and rankings of models fluctuate depending on evaluation conditions. Implementing decision-structured supervision enhances pivot-sensitive decision-making and medical QA within aligned knowledge limits, while a streamlined replay approach mitigates declines in overall assistant capabilities.
Key facts
- ClinPivot is an auditable treatment-decision benchmark built from biomedical relations and pivoted patient contexts.
- It tests whether models change treatment choices when new clinical constraints shift the action space.
- Strong medical QA performance does not reliably predict decision-making performance.
- Frontier models and task-adapted Qwen variants often fail to change decisions correctly.
- Model rankings shift across evaluation regimes.
- Decision-structured supervision improves pivot-sensitive decision-making and medical QA under matched knowledge budgets.
- Lightweight replay reduces losses in general assistant ability.
- The paper is submitted to arXiv under Computer Science > Artificial Intelligence.
Entities
Institutions
- arXiv