Can AI bots read your site?
Before a model can cite you, it has to be able to crawl your pages. Enter your domain and we'll check your robots.txt against the main AI bots. Free, no sign-up.
Every answer starts with a crawl
AI bots visit your site for three different reasons. Blocking one doesn't have the same effect as blocking another, so it pays to decide carefully.
Training
Bots such as GPTBot or ClaudeBot collect pages to train future models. Blocking them doesn't remove you from today's answers.
Search
Bots such as OAI-SearchBot or PerplexityBot index your site so it can be cited in web-search answers. Block them and you're unlikely to be cited.
User request
Bots such as ChatGPT-User or Claude-User open a page at the moment someone asks for it in a conversation.
Everything AI bots do on your site
The checker on this page is the first step. Inside the product we track the full picture.
Crawlability
Allowed, partial or blocked for each bot, and whether the rule is its own or inherited from the general rule.
Test a URL
Enter a path and see which robots.txt rule decides whether each bot can read it.
Bot visits
Connect your logs from Cloudflare, Vercel or another CDN, or through a webhook, and see which pages bots read and how often.
By platform and purpose
Visits split by company, by bot and by reason: training, search or user request.
Errors and status codes
Pages that return errors to AI bots, before they cost you citations.
- Coming soon
AI referrals
Connect Google Analytics 4 and measure the visits and conversions coming from ChatGPT, Perplexity and other assistants.
What bots read, page by page
With your logs connected you'll see which sections each bot visits, how often and with what result.
- Activity by hour, day, week or month
- Most-visited content by folder and URL
- Alerts when a bot starts getting errors
| Bot | Visits | Most-read section | Errors |
|---|---|---|---|
| GPTBot | 18.4K | /blog | 0.4% |
| OAI-SearchBot | 9.3K | /precios | 0.2% |
| ClaudeBot | 7.7K | /blog | 1.1% |
| PerplexityBot | 4.1K | /ayuda | 3.8% |
| ChatGPT-User | 2.3K | /productos | 0.0% |
Your visibility data, inside your agents
Connect Fedlum to Claude, ChatGPT or Cursor through our MCP server and ask in plain language. Or use the API to pipe the data into your own dashboards and workflows.
- Ask in Spanish or English, get answers from your data
- The same projects and permissions as in Fedlum
- An API for reports and automations
See where your brand stands today
Create your project, pick your prompts and get your first data in minutes. If you'd rather have us walk you through it, book a call.
7-day free trial · No card required
Questions we hear a lot
What is robots.txt?
A public file at the root of your site that tells each crawler which paths it may visit. The bots from the major AI companies say they respect it.
What does “partial” mean?
That the bot can read your site but some paths are blocked for it. Check they aren't pages you want AI to cite.
Do you store the domain I check?
No. The checker reads your robots.txt, evaluates it and shows you the result; we don't store the domain or the file.
Should I block training bots?
That's a business decision. What we do recommend is not blocking search and user bots by accident, because they're the ones that let you be cited today.