Welcome to the TechSEO Vitals Newsletter, brought to you by Martin Stepanek, an experienced technical SEO and web performance consultant with over 10 years of expertise in web development, website architecture design, and performance optimization.
Share
🟣 Your robots.txt Is Doing the Opposite of What You Think - TechSEO Vitals #35
Published 12 days ago • 6 min read
👋 Hey Reader, Welcome back to another episode of the TechSEO Vitals Newsletter! I see this on more sites than I’d like to admit. Someone wants to allow GPTBot, ClaudeBot, or PerplexityBot to crawl their site. So they add a clean Allow: / block for each one. Their wildcard rules already disallow /admin/, /cart/, /checkout/, and a few other paths they don’t want crawled. It looks correct. It reads correctly to a human. It’s completely wrong. GPTBot just got permission to crawl /admin/, /cart/,...
Subscribe to keep reading
This post is free to read but only available to subscribers.