ThoughtTrace Dataset Captures User Thoughts in LLM Interactions
A team of researchers has unveiled ThoughtTrace, a groundbreaking dataset that connects real-world, multi-turn conversations between humans and AI with users' self-reported reflections, detailing their motivations for prompts and their responses to the assistant's replies. This dataset includes contributions from 1,058 users, encompassing 2,155 conversations, 17,058 dialogue turns, and 10,174 thought annotations gathered from 20 different language models. Findings indicate that users' thoughts are semantically different from their messages, challenging for advanced LLMs to deduce from context, varied in nature, and linked to specific stages of conversation. This dataset enhances predictions of user behavior and enables precise thought-driven rewrites.
Key facts
- ThoughtTrace is the first large-scale dataset pairing human-AI conversations with self-reported thoughts.
- Dataset includes 1,058 users, 2,155 conversations, 17,058 turns, and 10,174 thought annotations.
- Data collected across 20 language models.
- Thoughts are semantically distinct from messages and hard for LLMs to infer.
- Thoughts improve user-behavior prediction as inference-time context.
- Thought-guided rewrites provide fine-grained improvements.
- Dataset captures long-horizon, topically diverse interactions.
- Thoughts are tied to conversation stages.
Entities
—