Sales, Marketing & SupportDone with custom-built or unnamed agent Oct 11, 2026
Audit cleaning-company sites for AI crawler access
An agent used MCP tools to read robots.txt rules for 14 AI crawlers and check 8 page tags on 18 cleaning-company sites; 9 returned their home page.
- What They Did
- The authors took domains with "cleaning" in the name from the Tranco top-1M list and filtered them down to 18 service firms. For each site they called two Site Check tools. One read robots.txt, headers and redirects for 14 named AI crawlers. The other checked 8 page tags. Site selection and filtering were done by hand from page titles.
- How It Went
- 9 of 18 sites returned their home page. Of those 9, one blocks 12 of the 14 crawlers, 7 lack a share image, and only 2 have all 8 tags. The other 9 sites gave a 403, a CDN error or no answer.
- Worth Knowing
- A refused or failed fetch doesn't show what AI bots receive, and firewall or CDN rules can block bots that robots.txt allows. Treat 9 sites as a snapshot, not a survey.
Try it yourself with custom-built or unnamed agent
Here is a list of [number] company websites: [list of domains]. For each one, fetch robots.txt and report whether it allows or blocks major AI crawlers such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended. Also check the homepage for meta tags that affect AI or search access, and flag any site that returns its homepage instead of a real robots.txt. Finish with a single table of results and a short list of the biggest gaps.
Do this with a skill.
These SkillShop playbooks do this exact job.