r/SEO • u/tjrobertson-seo • 11h ago
Discussion Cloudflare's pilot with OpenAI hints that ChatGPT's search index won't lean on links as much as Google does
Cloudflare and OpenAI put out a post this week about a pilot where Cloudflare's network helps OpenAI's search crawler decide what to crawl: https://blog.cloudflare.com/cloudflare-openai-discovery-agentic-web/
It doesn't say how ChatGPT's search algorithm actually works, but I think it drops a few real hints.
Most of it is about Cloudflare sharing data that helps OpenAI figure out which pages are worth indexing or updating, so the crawler spends less time on pages that aren't worth it or haven't changed and has more resources left to index more pages. That's aimed right at what's possibly Google's number one pain point right now. Billions of new pages keep getting added, Google is indexing a smaller and smaller share of them, and a lot of site owners are having a harder time getting pages into the index.
The part I keep coming back to is that the article specifically points out that a lot of the most helpful pages on the internet don't have many links pointing to them. My read is that whatever they're building won't depend on links as much as Google does. In Google's defense, they've said many times they don't want to rely on links as much as they do, and the algorithm does seem to lean on them less as it gets more sophisticated. I still think it depends on links more than they'd like to admit.
Whether that's enough to compete with Google, I honestly don't know. Google has a huge head start and a ton of data. On the other hand, Cloudflare is the biggest CDN and DNS provider, so it sees server-level data Google doesn't. And a search engine that only has to work for AI is a lot easier to build: people look at the top one to three results, while an AI will happily read through a hundred or more, so you mostly need to find the best one or two hundred pages rather than put them in perfect order.
I don't think ChatGPT is going to replace Google as the default search engine, but the competition does seem to be heating up.