AI Observatory Exposes Gaps in Company Reports on Real-World AI Use
The AI Observatory has been established by independent researchers to provide a counterpoint to data from AI firms such as Anthropic and OpenAI. This platform compiled 24,521 user interactions from seven datasets, involving 5,000 users and 52 models, including Claude and ChatGPT, during the period of 2023 to 2025. Co-directed by Anka Reuel and Shayne Longpre, the Observatory aims to offer unbiased evidence for AI decision-making. Notably, findings indicate that Anthropic's Economic Index excludes 48% of conversations that touch on more sensitive subjects. Model usage differs, with Grok concentrating on news, Claude on coding, and ChatGPT on academic assignments. However, the dataset remains modest compared to Anthropic's 1 million and OpenAI's 1.5 million conversations analyzed.
Key facts
- The AI Observatory aggregated 24,521 conversations, 85,633 turns, 5,000 users, and 52 models from seven datasets.
- Co-led by Anka Reuel (Stanford STAIR Lab) and Shayne Longpre (MIT Media Lab), with Data Provenance Initiative collaboration.
- Applying Anthropic's Economic Index methods to the Observatory's dataset would filter out 48% of conversations.
- Filtered-out conversations had higher levels of sensitive content: health/relationships 44.2% vs 31.2%, adult topics 7.9% vs 2.1%, harassment/hate 27.5% vs 5.66%, sexual content 16.7% vs 2.4%.
- Grok users skewed toward news and politics, with misinformation concentrating there; Claude for coding; Gemini for roleplay; ChatGPT for homework.
- Conversations with GPT-4o were longer and more iterative than with GPT-3.5, linked to reports of emotional addiction.
- Sensitive interactions decreased over time, suggesting improved platform safeguards; small talk increased while chatbot self-disclosure decreased.
- Anthropic Economic Index analyzed 1 million conversations and OpenAI's ChatGPT report analyzed 1.5 million, far exceeding the Observatory's 24,521.
- The Observatory's dataset is voluntarily provided, likely underrepresenting sensitive uses, so findings are not indicative of all AI use.
- Anthropic acknowledged the importance of independent external research; OpenAI and xAI did not respond to requests for comment.
Entities
Artists
- Anka Reuel
- Shayne Longpre
- David Widder
Institutions
- AI Observatory
- Stanford Trustworthy AI Research (STAIR) Lab
- MIT Media Lab
- Data Provenance Initiative
- Anthropic
- OpenAI
- xAI
- UT-Austin's School of Information