How do we know if our website is crawlable by AI systems?
Short answer
You test it, not assume it. Check robots.txt for the crawlers that matter, then fetch priority pages as those bots — through a crawler tool, curl with the right user agent, or server logs — and confirm three things: the response is 200, the raw HTML contains the actual content, and internal links expose every priority page. And remember there is no single "AI bot": search crawlers, training crawlers, and user-triggered fetchers behave differently and may be blocked in different places.
Expanded answer
Crawlability is a claim about behavior, so the proof is behavioral. robots.txt is the starting point — check which user agents are allowed — but it is only a request. The real answer comes from responses: fetch key pages with the relevant user agents and see what the server, CDN, and firewall actually return. A bot that gets 403s, challenges, or timeouts is blocked no matter what robots says.
Three families of crawlers matter, and they are not one bot. Search and answer-engine crawlers index content for search and AI answers. Training crawlers collect data for model training — some companies allow them, some deliberately do not; that is a policy choice, not a technical error. User-triggered fetchers open your page live when someone asks an assistant about you — blocking those silently removes you from live answers. Decide per family, then verify each one separately: allow-listing a single bot proves nothing about the rest.
Beyond access, confirm the content is readable: the raw HTML of priority pages contains the real text, not an empty JavaScript shell, and internal links expose every page you care about — a URL nothing links to is undiscoverable even for a welcome bot.
What to check
- robots.txt: which user agents are allowed or blocked, and whether that matches your actual policy.
- Real responses per user agent — status 200, no challenges, no error pages — via curl, a crawler tool, or logs.
- Server or CDN logs: which bots actually visit, what they receive, and what the firewall silently rejects.
- Raw HTML of priority pages contains the content itself.
- Priority pages are reachable through real internal links, not only search or menus.
Common mistakes
- Treating robots.txt as the whole answer while a firewall or bot-protection rule blocks the same crawlers.
- Allowing one famous bot and assuming "AI" can now read the site.
- Blocking user-triggered fetchers together with training crawlers and disappearing from live answers.
- Testing the homepage only, while priority pages sit behind blocked paths or missing links.
Not sure if your site is visible in AI-driven discovery?
Start with an AI Visibility Review.