Multimodal Model Enhances Repair Detection in Conversation
A new multimodal model has been developed by researchers to identify and categorize Other-initiated Repair (OIR) within conversational exchanges. This model integrates visual elements, including gaze shifts, facial expressions, body postures, and hand gestures. OIR occurs when a listener indicates an issue with speaking, hearing, or understanding, prompting the prior speaker to address the problem. Although conversation analysis has highlighted the role of both verbal and non-verbal cues in initiating OIR, previous computational methods primarily focused on text and audio data. The model's evaluation on two different corpora, featuring various languages and interaction contexts, shows that incorporating visual features enhances detection accuracy, a significant step for improving conversational agents' ability to resolve communication issues.
Key facts
- Other-initiated Repair (OIR) is an essential mechanism in conversational interaction.
- OIR involves a recipient signaling a problem in speaking, hearing, or understanding.
- Existing computational approaches for OIR detection rely mainly on text and audio.
- Conversation analysis shows OIR initiation includes non-verbal signals like gaze shifts and facial expressions.
- A novel multimodal model incorporates visual features for OIR detection and classification.
- The model was evaluated on two corpora with distinct languages and interaction settings.
- Results show visual features improve OIR detection.
- The work is relevant for conversational agents to handle communication breakdowns.
Entities
Institutions
- arXiv