Founder note (not a testimonial): I shipped this rollout on mentionpop.com because GEO measurement without crawl access is theater—you cannot cite a
/methodology/page models never fetch. This write-up documents infrastructure truth only, not marketing outcomes. — Aditya Kumar, Founder
Situation
MentionPop sells sampled AI visibility measurement—weekly GEO on ChatGPT and Gemini, Studio for Google AI Overviews, featured snippets, and PAA. Before content or prompts matter, public proof pages must be retrievable by AI crawlers.
In mid-2026, mentionpop.com sat behind Cloudflare with managed AI bot controls that effectively blocked several AI crawlers despite origin intentions to publish open marketing and research content.
That mismatch created a credibility gap: we talk about technical crawler access while our own edge configuration could deny fetches.
Problem definition
| Symptom | Risk |
|---|---|
| Managed block on AI bots | Stale or third-party summaries dominate |
| Managed robots rewriting | Origin robots.txt not served faithfully |
| Unclear validation | Teams assume allow without HTTP proof |
We needed technical access with dated evidence—not a press release claiming "10× visibility."
Changes implemented
1. Cloudflare AI bot posture
- Moved from blocking managed directives to allowing AI crawlers relevant to our public content strategy
- Disabled managed robots.txt rewriting so visitors and bots receive origin robots intent
2. Origin robots.txt clarity
Explicit allows for major AI agents on public marketing paths; disallows on authenticated app surfaces:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: *
Disallow: /dashboard/
Disallow: /account/
Disallow: /onboarding/3. Published llms.txt
Added machine-readable guidance so LLM crawlers locate canonical product facts:
# mentionpop.com/llms.txt (representative excerpt)
# MentionPop — B2B SaaS AI visibility measurement (GEO/AEO)
# NOT: chat UI mention pop-ups, Discord notifications, social listening
https://mentionpop.com/
https://mentionpop.com/methodology/
https://mentionpop.com/guides/
https://mentionpop.com/research/
https://mentionpop.com/about/llms.txt complements—not replaces—robots.txt.
Validation
On 2026-07-30, we ran a first-party baseline (full research note):
| Agent | robots.txt | HTTP fetch |
|---|---|---|
| GPTBot | Allowed | 200 |
| ClaudeBot | Allowed | 200 |
| PerplexityBot | Allowed | 200 |
| Google-Extended | Allowed | 200 |
- Sample size:
n=4 - Chart:
/resources/crawler-baseline-chart.svg - CSV:
/resources/ai-crawler-baseline-2026.csv
Scope reminder: this proves fetch access to public crawler-control files, not indexing, training inclusion, or recommendation in answers.
Results (honest)
What improved:
- Live robots.txt matches origin intent
- Direct user-agent requests return HTTP 200 on public control files
- Research and guide content is eligible for fetch by major AI bots
What did NOT change (by this case study's claims):
- No claimed mention rate lift in ChatGPT or Gemini
- No invented customer outcomes or testimonials
- No market-share statistics
Mention/citation performance continues to be tracked via sampled observation methodology—separate from this infrastructure note.
Separation of measurement and optimization
| Layer | This rollout | MentionPop weekly reports |
|---|---|---|
| Question | Can bots fetch our files? | Do sampled answers mention/cite us? |
| Evidence | HTTP 200 + robots.txt | Parsed provider responses |
| Failure handling | N/A | API failures excluded from scoring |
Read how to read a weekly GEO report for reporting cadence.
Lessons for other teams
- Check the CDN first — origin robots.txt is not the whole story on Cloudflare.
- Validate with user-agent fetches — do not trust dashboard toggles alone.
- Publish dated baselines —
n=4CSV beats a slide that says "bots allowed." - Do not sell access as visibility — pair crawl fixes with recurring GEO measurement.
Rollback and monitoring plan
Infrastructure changes without rollback plans create silent regressions. For the Cloudflare rollout we documented:
| Checkpoint | Action if failed |
|---|---|
| Post-change robots.txt | Revert CDN bot policy; restore prior snapshot |
GPTBot fetch to /methodology/ |
Open engineering ticket; block public "access fixed" messaging |
| llms.txt HTTP status | Fix routing before announcing research index |
| Weekly GEO sample (separate) | Continue reporting; do not attribute mention deltas to CDN |
We re-run the four-agent baseline after any CDN or robots.txt edit—not only on launch day. The CSV format in /resources/ai-crawler-baseline-2026.csv is the template for future dated rows.
Timeline (first-party)
| Phase | Work |
|---|---|
| Discovery | Identified managed block conflicting with origin allow intent |
| Change window | Switched AI bot posture to allow; disabled robots rewriting |
| Origin updates | Published explicit agent allows + app disallows |
| llms.txt | Added canonical paths for product, methodology, guides, research |
| Validation | n=4 agent baseline on 2026-07-30 with chart + CSV export |
| Measurement | Resumed weekly ChatGPT/Gemini samples without claiming lift from CDN |
This timeline is infrastructure truth. Mention/citation trends are interpreted only through sampled observation methodology on their own schedule.
Infrastructure work is unglamorous GEO—and founder-note honest case studies age better than fabricated lift charts.
Frequently asked questions
Did this rollout increase MentionPop mentions in ChatGPT?+
This case study documents infrastructure access only. We do not claim a visibility lift tied to the Cloudflare change—mention/citation outcomes are measured separately via sampled provider runs.
What changed in Cloudflare?+
Managed AI bot blocking directives were replaced with explicit allow posture and managed robots.txt rewriting was disabled so origin robots.txt intent is served faithfully.