Guideline-as-Oracle: Zero-Annotation Training of an Ophthalmic Telephone Triage Agent
There's a new AI training method called Guideline-as-Oracle (GAO) that takes guidance from the American Academy of Ophthalmology and creates a detailed 70-row rule table. This table serves as the main source for training 3,000 dialogues, with human input only used for evaluation. It addresses the tough issues of scaling supervision for complex medical conversations, which can be costly and raise privacy concerns. The researchers propose eight dialogue construction strategies and assess their effectiveness. After fine-tuning a model with 9 billion parameters on this data, they developed GAO-Triage, improving agreement with a reference set from 61.7% to 74.1% and recall from 9% to 100%. You can check out their findings on arXiv under the identifier 2608.04772.
Key facts
- Guideline-as-Oracle (GAO) compiles American Academy of Ophthalmology guidance into a 70-row operational rule table.
- GAO uses the rule table as the sole source of instance-level supervision for 3,000 training dialogues.
- Human labeling is reserved for evaluation only.
- Eight construction strategies are cataloged, including cited-row tier assignment, one-fact boundary pairs, metadata-only repair, and label repair.
- Each strategy's evidential status is characterized as labeling mechanism, null, confounded, or evaluated only as a package.
- Fine-tuning a 9B backbone on the corpus yields GAO-Triage.
- GAO-Triage improves agreement with a 201-case operational reference from 61.7% to 74.1% (exact McNemar p=0.0046).
- Emergent-case recall improves from 9% to 100%.
Entities
Institutions
- American Academy of Ophthalmology
- arXiv