How to check whether AI crawlers can access your website

If AI platforms cannot crawl your pages, your content will not appear in AI-generated answers, citations, or generative summaries.

Over the past two years, tools like ChatGPT Search, Perplexity, and Claude have evolved from offline language models into real-time answer engines. They regularly fetch web pages to cite primary sources, answer user queries, and build training datasets.

In my work on LLM optimisation, ensuring crawl accessibility sits directly at step 2 of my 5-step process: web platform accessibility. Here is how to verify whether AI bots can actually fetch, read, and understand your website.

Which AI crawlers should you check?

AI companies run different bots for different tasks. Most operate separate crawlers for training their foundation models and for real-time search retrieval.

If you block a training bot, you stop an organisation using your writing to train future models. If you block a search bot, you remove your brand from real-time AI responses and citations. Knowing which bot does what helps you make deliberate choices rather than locking out valuable search traffic by accident.

Bot user agent	Operator	Primary purpose	Official documentation
OAI-SearchBot	OpenAI	Real-time search indexing for ChatGPT Search	OpenAI crawler docs
GPTBot	OpenAI	Data collection for AI model training	OpenAI crawler docs
ChatGPT-User	OpenAI	Live web fetches triggered directly by users	OpenAI crawler docs
ClaudeBot	Anthropic	Content collection and training for Claude	Anthropic crawler docs
PerplexityBot	Perplexity	Real-time web indexation and citation search	Perplexity crawler docs
Google-Extended	Google	Token training control for Gemini and Vertex AI	Google Search Central
Applebot-Extended	Apple	Training control for Apple Intelligence	Apple support docs

Google-Extended is a control token for training, not a distinct web crawler. Google still uses standard Googlebot to fetch your pages.

How to check your robots.txt file

The quickest way to check access is your robots.txt file, located at https://yourdomain.com/robots.txt.

Open the file in your browser and look for your Disallow rules. Many developers and CMS plugins automatically block crawlers they do not recognise.

Check for directives like these:

User-agent: *
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

If your file has Disallow: / under User-agent: *, every bot that obeys robots.txt will ignore your entire website unless granted an explicit exception higher up.

To give search bots access while retaining control over training data, your configuration should be explicit. For example, if you want to allow ChatGPT Search citations while blocking training crawlers, you would configure:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Verify that none of your key landing pages, category pages, or product resources sit behind disallowed paths like /products/ or /insights/.

How to inspect firewalls and CDN bot management

A clean robots.txt does not guarantee an AI bot can reach your server. Security rules and Web Application Firewalls (WAFs) frequently block AI user agents before they ever touch your application.

Cloudflare, AWS WAF, Akamai, and Imperva have all introduced automated switches to block AI crawlers. For instance, Cloudflare provides a one-click setting to block AI scrapers. If someone on your engineering team turned this on, it might block both scrapers and legitimate search bots like OAI-SearchBot.

[Joe: add a real example here, e.g. a client whose CDN or WAF was silently blocking AI search bots]

To diagnose firewall interference:

1. Check your CDN dashboard. Look at the security analytics or event logs for HTTP 403 (Forbidden) and HTTP 429 (Too Many Requests) responses.
2. Filter firewall logs by User-Agent strings such as OAI-SearchBot or ClaudeBot.
3. Check Managed Rulesets. Look for rules categorised under "Bot Management", "AI Crawlers", or "Automated Scrapers".
4. Whitelist legitimate bots. If your WAF categorises search bots as threats, create an explicit firewall allow-rule based on verified bot identity.

How to verify AI visits in server logs

Server logs are the ultimate source of truth. They record every single request that reaches your web server, complete with response codes and timestamps.

Ask your hosting provider or engineering team for your raw access logs. You can search the log files for specific crawler user agents using basic terminal commands or log analysis tools.

For instance, you can run a quick command on Linux or macOS servers:

grep -i "OAI-SearchBot" access.log | tail -n 20

When you examine the output, check the HTTP status code returned to the crawler:

• 200 OK: The crawler accessed your page successfully.
• 301 or 302: The bot was redirected. Ensure redirects do not loop or point to blocked destinations.
• 403 Forbidden: A firewall or server rule refused access.
• 429 Too Many Requests: Your server is rate-limiting the crawler because of aggressive crawl speed.
• 500 or 503: Your server failed while handling the request.

Verifying legitimate bot IPs

Bad actors often fake their user agent to impersonate official bots. Legitimate AI providers publish their IP ranges or support reverse DNS lookups.

To verify a request genuinely came from an OpenAI crawler, run a reverse DNS lookup on the IP address in your terminal:

host <IP-ADDRESS>

A genuine OpenAI bot will resolve back to an official domain ending in openai.com or searchbot.openai.com.

Why JavaScript rendering can block AI crawlers

Even if a crawler receives a 200 OK response, it may still be unable to read your content if your website relies heavily on client-side JavaScript.

Standard search engines like Google have dedicated rendering pipelines that run JavaScript before indexing text. However, many AI crawlers do not execute JavaScript at all. They fetch only the raw HTML sent in the initial HTTP response. If your website is built on a client-side Single Page Application (such as standard React, Angular, or Vue) without server-side rendering, an AI crawler will only see an empty shell.

Test this by viewing your page source:

1. Right-click on your page in your browser and select View Page Source (do not use Inspect Element).
2. Use Ctrl + F (or Cmd + F) to search for your key text, headings, or product pricing.
3. If your content is absent from the source code, AI search bots cannot read it.

To fix this, implement server-side rendering (SSR), static site generation (SSG), or dynamic rendering to ensure the complete textual content is present in the initial HTML payload.

How to start checking your AI crawler access

You do not need to overhaul your technical infrastructure in a single afternoon. Follow these practical steps to audit your access today:

1. Review robots.txt: Confirm you are not accidentally disallowing crawlers like OAI-SearchBot or PerplexityBot.
2. Review your WAF configuration: Ensure your Cloudflare, Akamai, or AWS security settings do not drop AI search traffic silently.
3. Inspect your raw HTML: Ensure that primary body content appears without requiring JavaScript execution.
4. Analyse your access logs: Confirm whether bots are returning HTTP 200 codes or running into 403 or 429 errors.

If you suspect your website is invisible to AI platforms or if you want to audit your complete crawl architecture, get in touch for a detailed technical review.