This research note documents a first-party, point-in-time baseline of AI crawler access on mentionpop.com as of 2026-07-30. It exists because GEO conversations often conflate three unlike claims:
- Technical fetch access (can the bot retrieve public files?)
- Search indexing (is content in a search index?)
- Generative recommendation (does a model mention or cite you?)
This study addresses only #1.
Scope and limitations
| In scope | Out of scope |
|---|---|
Live robots.txt allow/disallow for four agents |
Mention/citation rates in ChatGPT or Gemini |
| HTTP status on direct fetches with user-agent headers | Consumer UI behavior |
Public crawler-control files (robots.txt, llms.txt) |
Third-party domains |
| Dated reproducibility on mentionpop.com | Market-wide crawler statistics |
Sample size: n=4 agents — GPTBot, ClaudeBot, PerplexityBot, Google-Extended.
Interpretation guardrail: "Allowed + HTTP 200" means the bot could retrieve public control files at test time. It does not prove favorable treatment in AI answers.
Methods
1. robots.txt inspection
We fetched the live robots.txt served at https://mentionpop.com/robots.txt without authentication and recorded explicit rules for:
GPTBot(OpenAI)ClaudeBot(Anthropic)PerplexityBot(Perplexity)Google-Extended(Google generative use policy discussions)
2. User-agent fetch verification
For each agent, we issued direct HTTP requests to public crawler-control paths using the corresponding user-agent string, recording response codes. Targets included robots.txt and published llms.txt.
3. CDN configuration context
mentionpop.com is fronted by Cloudflare. Managed AI bot settings previously interfered with origin robots intent; configuration was updated to allow crawlers and disable managed robots rewriting—details in the Cloudflare rollout case study.
Results (2026-07-30)
| Agent | robots.txt | HTTP status | Notes |
|---|---|---|---|
| GPTBot | Allowed | 200 | Public files retrievable |
| ClaudeBot | Allowed | 200 | Public files retrievable |
| PerplexityBot | Allowed | 200 | Public files retrievable |
| Google-Extended | Allowed | 200 | Public files retrievable |
Visual summary: /resources/crawler-baseline-chart.svg (1200×630).
Machine-readable export: /resources/ai-crawler-baseline-2026.csv.
Example robots.txt excerpt (illustrative)
Live files evolve—always verify on your domain:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /Private app routes remain disallowed separately (/dashboard/, /account/, etc.).
Relationship to MentionPop product measurement
MentionPop weekly recurring GEO observes ChatGPT and Gemini for mentions and citations via provider-sampled API runs—not by replaying this crawler test.
- Studio adds Google AI Overviews, featured snippets, and PAA.
- Perplexity Sonar is used for selected internal validation, not default recurring monitoring.
- Failed API calls are excluded rather than scored as misses.
Crawler access and answer observation are complementary evidence layers. Improve access with technical crawler access guidance; measure answers with sampled observation methodology.
Replication checklist
Teams replicating this baseline on their domains should:
- Record date, time, timezone
- Save raw HTTP headers and status codes
- Note CDN bot management settings
- Store robots.txt and llms.txt snapshots
- Avoid claiming visibility lifts from access alone
Historical context (why we published this)
MentionPop sells measurement—not crawl consulting. We published this baseline because credibility requires practicing what we document. If our own edge blocked GPTBot while guides advocated allow rules, customer trust would erode faster than any single mention delta.
Comparison to third-party crawler studies
Industry blog posts sometimes aggregate dozens of domains without reproducible methods. This note intentionally scopes to:
- One domain we control (mentionpop.com)
- Four agents with logged fetch metadata
- One date (2026-07-30)
Readers should not extrapolate market-wide "AI crawl openness" from n=1 site × n=4 agents.
Future work (explicit non-claims)
Potential extensions—not included in this release:
- Repeated weekly fetches to measure drift
- Path-level allow rules beyond root robots.txt
- Correlation studies linking fetch access to mention/citation rates (requires separate experimental design)
Any future correlation study must avoid causation hype and disclose prompt/sample design upfront.
Data dictionary (CSV)
| Column | Meaning |
|---|---|
agent |
User-agent token tested |
robots_txt_status |
Allow/disallow as read from live robots.txt |
http_status |
Status code on direct fetch with agent header |
tested_at |
ISO date of run |
scope_note |
Reminder that access ≠ recommendation |
Download: /resources/ai-crawler-baseline-2026.csv.
llms.txt validation (same test window)
Alongside robots.txt, we fetched https://mentionpop.com/llms.txt with the same four user-agent strings. All returned HTTP 200 with plain-text guidance listing canonical product, methodology, and content hub paths.
llms.txt does not override robots rules—it helps crawlers locate stable facts after fetch is permitted. Teams publishing llms.txt should verify:
- File is reachable without authentication
- Listed URLs themselves return 200 for the same agents
- Negative definitions ("what we are not") match visible
/about/copy
See entity disambiguation for AI visibility for why negative definitions matter in policy files.
This baseline will age. Re-run after infrastructure changes and publish dated deltas—GEO credibility compounds when evidence does.
Frequently asked questions
Does this prove MentionPop is recommended in ChatGPT?+
No. This baseline proves technical fetch access to public crawler-control files—not indexing, training inclusion, or mention/citation outcomes in generative answers.
Why only four agents?+
This study scopes to the four agents verified on mentionpop.com on the test date. Additional bots require separate dated runs—do not extrapolate beyond n=4.
How is this related to GEO measurement?+
Crawler access is a prerequisite for fresh primary HTML to be available to generative systems. MentionPop separately measures sampled mentions and citations via provider APIs.