๐ Hey Reader, Lighthouse shipped a new agentic audit a few weeks ago. One of the things it checks is whether you have an llms.txt file. The same month, Google's own guidance on AI search listed llms.txt under the things you don't need to do. One company. Telling you to build the file, and telling you not to bother with it. Same month. No wonder everyone's confused. The confusion clears the moment you stop treating llms.txt as a search thing. It was never a search thing. Here's where I land....
17 days agoย โขย 6 min read
๐ Hey Reader, You submit 48K URLs in your XML sitemap. 11K get indexed. You keep asking why Google won't crawl more. You have it backwards. Google crawled plenty. It just disagreed with your definition of "worth indexing" on 77% of what you submitted. The pattern is almost always the same. Sitemaps full of pages nobody would defend in a content review, submitted on autopilot, signaling to Google that the overall quality of your site is weak. You're probably treating your sitemap as a list. A...
about 1 month agoย โขย 6 min read
๐ Hey Reader, Welcome back to another episode of the TechSEO Vitals Newsletter! Iโve had this conversation three times in the past month. A site ranks fine in Google. Decent traffic, decent positions. Then someone runs a few brand prompts in ChatGPT or Claude and gets nothing. Or worse, gets a clean citation for the competitor down the road. The site owner is confused. Weโre #2 in Google for that query. How are we invisible here? The answer is almost always the same. The site has been getting...
about 1 month agoย โขย 5 min read
๐ Hey Reader, Welcome back to another episode of the TechSEO Vitals Newsletter! I see this on more sites than Iโd like to admit. Someone wants to allow GPTBot, ClaudeBot, or PerplexityBot to crawl their site. So they add a clean Allow: / block for each one. Their wildcard rules already disallow /admin/, /cart/, /checkout/, and a few other paths they donโt want crawled. It looks correct. It reads correctly to a human. Itโs completely wrong. GPTBot just got permission to crawl /admin/, /cart/,...
about 2 months agoย โขย 6 min read
๐ Hey Reader, Welcome back to another episode of the TechSEO Vitals Newsletter! Every time ChatGPT gives up on one of your pages, it leaves a trace in your server logs. Almost nobody looks for it. It might be the best signal you have for whether AI tools can actually read your content. โก๏ธ TL;DR A 499 error means a bot (or visitor) closed the connection before your server finished responding ChatGPT-User has a brutally short patience window and dominates 499s on most sites The bot gets nothing...
2 months agoย โขย 5 min read
๐ Hey Reader, Welcome back to another episode of the TechSEO Vitals Newsletter! Cloudflare is one of the most popular services for speeding up and protecting websites. Over 20% of all internet traffic flows through their servers. That alone makes it worth talking about the problems they can cause when used without proper knowledge. Iโm a huge fan of Cloudflare. They offer some of the best tools on the market, many of them completely free. But today I want to focus on one specific area: their...
3 months agoย โขย 5 min read
๐ Hey Reader, Welcome back to another episode of the TechSEO Vitals Newsletter! Every website carries technical debt. Some of it started years ago. Some of it was there from day one. This is especially valid for enterprise sites with thousands of pages, where the consequences are brutal. I see it constantly. A company asks me to help with their Core Web Vitals. Within minutes, itโs clear the problems go much deeper. The foundation itself is broken. Technical debt works the same way as...
3 months agoย โขย 4 min read
๐ Hey Reader, Welcome back to another episode of the TechSEO Vitals Newsletter! The golden era when Google adapted to our slow, badly coded websites is over. Google is still patient, but dozens of AI crawlers play a completely different game. These crawlers behave differently than Googlebot. If your website is slow, they wonโt wait. Theyโll leave, and your content will never make it into ChatGPT, Claude, Perplexity, or other AI tools. โก๏ธ TL;DR AI crawlers make one request with no JavaScript...
4 months agoย โขย 4 min read
๐ Hey Reader, Welcome back to another episode of the TechSEO Vitals Newsletter! The last few months have been confusing. What should we do with our websites to prepare them for AI? What should we avoid? Finding the right answers is harder than ever because multiple authorities are saying the opposite things. To figure out what actually improves visibility across AI search and tools, we need to go deeper. We need to understand how AI models work, what server logs reveal about crawler behavior,...
4 months agoย โขย 4 min read