🟣 Your robots.txt Is Doing the Opposite of What You Think - TechSEO Vitals #35


👋 Hey Reader, Welcome back to another episode of the TechSEO Vitals Newsletter! I see this on more sites than I’d like to admit. Someone wants to allow GPTBot, ClaudeBot, or PerplexityBot to crawl their site. So they add a clean Allow: / block for each one. Their wildcard rules already disallow /admin/, /cart/, /checkout/, and a few other paths they don’t want crawled. It looks correct. It reads correctly to a human. It’s completely wrong. GPTBot just got permission to crawl /admin/, /cart/,...