A study conducted by the University of Duisburg-Essen and the Max Planck Institute for Software Systems delves into the differences between traditional search engines and generative AI systems in terms of information sources and presentation. The research team compared Google's natural search results with four generative AI search systems, including Google AI Overview, Gemini 2.5 Flash with search functionality, GPT-4o-Search, and GPT-4o with search tools. This study involved over 4,600 queries across six topics, including politics, product reviews, and science, revealing the diversity of these systems in processing online information.
Image source note: The image is AI-generated, and the image licensing service is Midjourney
The study found that AI search systems can retrieve information from a broader and less well-known range of websites. For example, 53% of the cited websites in the AI Overview did not appear in the top ten results of Google search, while 27% did not even make it to the top 100. This suggests that users may see information from less reviewed or unfamiliar websites.
In terms of the depth of reference to information sources, GPT-Tool cited an average of only 0.4 external sources per answer, mainly relying on its internal data. In contrast, AI Overview and Gemini cited more than eight external sources per query, demonstrating their richness in information retrieval. Additionally, traditional Google search typically limits results to ten links per query, making its information sources more concentrated.
The study also emphasized that traditional search engines outperformed AI systems in handling current events. In testing 100 hot topics, AI Overview could cover only 3% of the queries, while GPT-4o-Search covered 72%. This highlights the shortcomings of AI systems in updating the latest information, which may lead to users receiving outdated or inaccurate information.
Key points:
🌐 53% of the websites referenced by AI search systems do not appear in the top ten results of Google, showing a more diverse source.
📊 GPT-Tool cites an average of only 0.4 external sources, mainly relying on internal knowledge.
📰 Traditional search engines are significantly more accurate than AI search systems in handling current events.
