CallBench: Benchmark for Dual-Goal Phone Call Assistants
A team of researchers has unveiled CallBench, a benchmark designed to assess dual-goal coordination in phone call assistants, specifically tailored for the Chinese language. This benchmark features 50,000 fully realized multi-turn dialogues spanning six distinct scenarios: takeout, delivery, taxi, work, life, and harassment. It encompasses standard presets, emergent presets, and cases without presets, showcasing various relationships between goals from both the owner and caller, including alignment, complementarity, irrelevance, and conflict. Additionally, the study introduces a turn-level evaluation protocol that is aware of presets, focusing on semantic comprehension and context. The research paper can be accessed on arXiv under ID 2607.22635.
Key facts
- CallBench is a Chinese benchmark for dual-goal coordination in phone call assistants.
- It contains 50,000 multi-turn phone call dialogues.
- Dialogues span six scenarios: takeout, delivery, taxi, work, life, and harassment.
- Covers regular presets, emergent presets, and no-preset cases.
- Includes relations: alignment, complementarity, irrelevance, and conflict.
- Features a preset-aware turn-level evaluation protocol.
- Paper available on arXiv with ID 2607.22635.
Entities
Institutions
- arXiv