Study Reveals Unreliable Multi-Turn Behavior in LLMs Through Repair Analysis

A recent study investigates how large language models handle repairs in multi-turn conversations, particularly with solvable and unsolvable math problems. It assesses whether these models autonomously initiate repairs and how they react to user-driven repair efforts. Findings reveal notable variations among models, with some showing almost total resistance to repair attempts while others are easily influenced. As dialogues progress beyond a single turn, the behavior of the models becomes increasingly unique and unpredictable. Each language model tested demonstrates its own specific type of unreliability concerning repairs. This research, published on arXiv, underscores that while repair is crucial for resolving issues in human conversations, it remains insufficiently studied in human-LLM interactions.