GuideShould you block AI crawlers?
Thousands of businesses blocked GPTBot and other AI crawlers in robots.txt back when the debate was about training data — and never revisited it. In 2026 the calculus changed: assistants browse the live web to answer buying questions, and a blocked crawler means the AI literally cannot read your site when a buyer asks about you. Many sites are invisible to AI by accident, not by choice.
FAQ
What happens if my robots.txt blocks GPTBot or ClaudeBot?When a buyer asks ChatGPT or Claude about your category, the assistant can't fetch your pages — so it recommends a competitor it can read. You inherit an invisibility penalty from a decision made years ago for a different reason.
Should anyone still block AI crawlers?Publishers whose entire product is their content sometimes choose to. But for any business that wants customers to find it — local services, SaaS, e-commerce — blocking the crawlers that answer buying questions is self-sabotage.
How do I check what I'm blocking?Run a free Cited scan — it reads your robots.txt, flags blocked AI crawlers, and shows whether each major engine can actually see and recommend your site.
See if AI recommends you — free
Most businesses score under 40/100 and have no idea. Find your score in 15 seconds, then fix it.
Run my free scan