A recent report by content licensing platform TollBit has exposed a significant disparity in referral traffic sent to publishers by AI search engines like those from OpenAI and Perplexity compared to traditional search engines like Google.
Forbes reports that the rapid rise of AI-powered search engines has led to bold claims by companies like OpenAI and Perplexity, suggesting that their summarized answers, generated by scraping information from the web, would benefit publishers by directing more readers to their sites. However, a new report shared exclusively with Forbes paints a starkly different picture.
According to the report by TollBit, a content licensing platform, AI search engines send 96 percent less referral traffic to news sites and blogs compared to Google search. This revelation comes as a blow to publishers who had hoped that the emergence of AI search would bring a new source of income.
The report, which analyzed 160 websites across various categories, including national and local news, consumer tech, and shopping blogs, found that AI developers’ scraping of websites has more than doubled in recent months. On average, OpenAI, Perplexity, Meta, and other AI companies scraped websites 2 million times in the fourth quarter of 2024, with each page being scraped about seven times on average.
Toshit Panigrahi, CEO of TollBit, emphasized the significant demand for publisher content, stating, “We are seeing an influx of bots that are hammering these sites every time a user asks a question.”
The impact of AI search on businesses relying on search traffic is already being felt. Edtech company Chegg recently filed a lawsuit against Google, alleging that the search giant’s AI-generated summaries included content from its website without attribution, resulting in a significant decline in traffic and revenue. Chegg’s CEO, Nathan Schultz, expressed concerns about the broken social contract between Google and publishers, which traditionally involved sending users to high-quality content rather than retaining traffic on Google.
Travel booking sites like Kayak and TripAdvisor have also raised concerns about Google’s AI search overviews chipping away at their traffic. Meanwhile, news publishers have taken legal action against OpenAI and Perplexity for alleged intellectual property infringement.
The report also highlighted the difficulty in identifying and understanding how AI companies access website content, as many do not properly disclose their scraper bots. This lack of transparency makes it challenging for website owners to block unwanted scraping without potentially impacting their search engine optimization (SEO).
Perplexity, a $9 billion-valued AI search startup, has faced criticism for scraping and republishing paywalled articles from various news outlets without adequate attribution. The company has been sued by the New York Post and Dow Jones for alleged copyright infringement and attributing made-up facts to media companies.
In addition to the impact on search traffic and revenue, rampant AI scraping is also causing publishers to incur millions in server costs as bots visit websites to read and scrape content. With the launch of research AI agents by companies like OpenAI and Perplexity, which autonomously visit hundreds of sites to produce in-depth reports, this problem is expected to worsen.
Read more at Forbes here.
Lucas Nolan is a reporter for Breitbart News covering issues of free speech and online censorship.
