5.9% of sites block AI bots accidentally
45sThe surprising statistic and common mistake create curiosity and urgency for website owners to check their own settings.
▶ Play ClipAI Summary
Full Transcript
About 5.9% of websites are accidentally blocking ChatGPT from crawling them. That's millions of sites invisible to AI search, and it's probably not on purpose. A lot of sites inherit robots.txt rules [music] from templates. Like, if you're on Cloudflare, they have a feature that's now on by default that tells AI bots to stay away from your content. So, here's a quick tip to check if you're blocking ChatGPT and other AI search bots. Go to yourdomain.com/robots.txt. Look for
GPT bot, Claude bot, Google extended, or OAI search bot. If you see disallow next to any of those, you're asking that AI platform not to read your site, which means it probably won't recommend that page.