CARE-MH Framework Aims to Standardize Mental Health LLM Evaluation
A new framework called CARE-MH has been developed by researchers to facilitate consistent and reproducible assessments of large language models (LLMs) in the realm of mental health support. This framework tackles the issues present in current benchmarks, which often suffer from inconsistent evaluation designs and metric definitions, making comparison and reproduction challenging. The application of CARE-MH to reproduce and scrutinize leading benchmarks indicates that model stability significantly influences reproducibility, while discrepancies between benchmarks mainly stem from varied metric definitions. These results highlight the importance of establishing standardized evaluation setups and unified metric definitions for upcoming mental health LLM benchmarks.
Key facts
- CARE-MH is a unified framework for evaluating mental health LLMs.
- Existing mental health benchmarks suffer from inconsistent evaluation designs and metric definitions.
- Reproducibility depends strongly on model stability.
- Cross-benchmark disagreement primarily arises from differences in metric definitions.
- The study calls for standardized evaluation configurations and shared metric definitions.
- LLMs are increasingly used to provide mental health support.
- Evaluation must cover safety, empathy, and therapeutic appropriateness.
- The framework aims to make evaluations comparable and reproducible.
Entities
Institutions
- arXiv