Study Reveals Generative AI Disrupts Google Search Results
A recent empirical analysis published on arXiv examines the impact of generative AI on web search, specifically how it modifies information retrieval and presentation. The study presents a publicly available benchmark dataset comprising 11,500 user queries to evaluate outcomes from Google's standard search engine, its AI Overview (AIO), and Gemini Flash 2.5. Notably, the research reveals that AIOs appear above organic search results for 51.5% of typical user queries, particularly for contentious topics. Furthermore, the sources retrieved by the three search engines show considerable divergence, with an average Jaccard similarity of less than 0.2, highlighting that traditional Google search maintains a stronger connection to organic results compared to AI-generated summaries.
Key facts
- Generative AI is increasingly integrated into web search.
- Study introduces a benchmark dataset of 11,500 user queries.
- Compares Google search, AI Overview (AIO), and Gemini Flash 2.5.
- 51.5% of real-user queries generate an AIO.
- AIOs are displayed above organic search results.
- Controversial questions frequently result in an AIO.
- Retrieved sources have less than 0.2 average Jaccard similarity.
- Traditional Google search is more linked to organic results.
Entities
Institutions
- arXiv