Statistical Framework for Reliable Multi-Agent LLM Self-Harm Screening
A newly developed statistical framework for multi-agent LLM pipelines seeks to enhance the reliability of self-harm risk assessments. This system is designed as directed acyclic graphs (DAGs) and provides adaptive decision-making, moving away from traditional heuristic voting methods. Each agent is represented as a stochastic categorical decision, which allows for more precise confidence bounds on agent performance. It also incorporates a bandit-based adaptive sampling strategy that adjusts based on input difficulty, along with regret guarantees that demonstrate logarithmic error growth during deployment. This method addresses the shortcomings of typical evaluation techniques like LLM-as-a-judge, which fail to reflect decision reliability or error accumulation, vital in safety-sensitive behavioral health contexts. The framework was tested on two labeled datasets.
Key facts
- The framework is designed for multi-agent LLM pipelines in behavioral health and psychiatry.
- Tasks include assessing self-harm risk and screening for depression.
- The pipeline is structured as directed acyclic graphs (DAGs).
- It provides an alternative to heuristic voting with principled, adaptive decision-making.
- Each agent is modeled as a stochastic categorical decision.
- Three innovations: tighter agent-level confidence bounds, bandit-based adaptive sampling, and regret guarantees.
- Regret guarantees show logarithmic error growth when deployed.
- Evaluated on two labeled datasets.
Entities
—