Track AI Bots in Logs
Learn how to identify AI crawlers, review status codes, and spot access patterns in server logs.
Core guides and experiments on GEO, AEO, AI crawling, schema, and search visibility.
Before AI changed search, SEO followed a very different set of rules — and a very different goal.
You could build an amazing website, but if no one knew it existed, there wasn’t much point.
Early on, the approach was simple. People put their website everywhere they could:
That helped — and it still does.
But as the web grew, that approach stopped being enough.
The web was still being built. I like to think that’s part of why it was called “The web.” It really did feel like a spider web — everything was connected, but the connections didn’t feel very sturdy.
Visitors might find your site, but:
It was a bit of a wild west time.
Personally, I found a lot of information through newsgroups. That might have been because I was a bit of a nerd and was looking for exactly the kind of discussions happening in those spaces — newsgroups, forums, and IRC channels.
Search engines weren’t the center of discovery yet. People found information through communities.
“Power Macintosh 9600/350 back side” by Gona.eu, Wikimedia Commons, licensed under CC BY-SA 3.0
Suddenly, websites were actually being indexed.
If you wanted to learn something — PHP, how to build a database, how to connect it all — the information was everywhere. People weren’t just talking about it, they were showing you exactly how to do it.
A lot of the internet was free.
The biggest cost wasn’t content — it was time. You were paying for access, often by the hour.
And it was slow.
A lot of people were still connecting through a phone line, listening to the modem dial and negotiate a connection. Once you heard that familiar handshake, you knew you were in.
It wasn’t perfect, but for the first time, it felt like you could actually find things.
As the web grew, it became impossible to rely on directories, communities, or word of mouth alone. There was simply too much content.
Search engines became the gateway.
Instead of asking, “Where do I find this?” people started asking, “Can I search for this?” And that changed everything.
Before AI, search engines worked in a much more literal way. They weren’t trying to fully understand meaning — they were trying to match patterns.
If you knew what signals they were looking for, you could influence the outcome.
And for a long time, that worked surprisingly well.
SEO was more mechanical than it is today.
The focus was on:
If you could align those elements correctly, you could rank.
It wasn’t perfect, but it was predictable.
Search engines evolved. Google refined how it ranked pages, filtered out spam, and rewarded higher-quality content.
SEO matured along with it.
We learned how to:
It wasn’t static, but it was understandable.
If you paid attention, tested, and adapted, you could still win.
At first, it was subtle.
Better query understanding. Smarter results. More relevant answers.
But eventually, something bigger happened.
Search stopped just pointing to answers — it started becoming the answer.
As of this writing in April 2026, it has been a bit of a wild ride.
A lot of people in SEO had to stop and ask:
Because the rules didn’t just change — the outcome changed.
Across every major search engine, AI-generated answers became part of the experience.
And that changed user behavior almost overnight.
People weren’t always clicking through anymore.
They were getting what they needed directly from the results.
You could still rank.
Your content could still be correct, useful, and technically sound.
But the user might never visit your site.
They would read the answer, make a decision, and move on.
Sometimes they would:
…without ever clicking a single link.
Someone has a plumbing issue.
They search.
AI summarizes:
And that’s enough.
They call.
They never visit the website.
Because now:
Even analytics started to feel less reliable.
You could do everything right and still struggle to show what happened.
No.
It meant you had to be much more intentional about how you built and organized it.
You couldn’t just put up a page and expect it to perform.
You had to actually work at it.
Simple pages filled with generic claims like:
stopped carrying any real weight.
That kind of content became what it really was: unauthoritative fluff.
If AI couldn’t trust it, it wouldn’t use it.
What worked before was often about being seen.
What works now is about being understood.
SEO didn’t disappear.
The same core mechanics still exist:
But the outcome has changed.
Users aren’t always clicking through a list of links anymore. They’re getting answers directly — summarized, rewritten, and combined from multiple sources.
That means ranking alone isn’t enough.
A page can rank and still never be seen.
What matters now is whether your content is:
This is where modern SEO shifts:
SEO isn’t dead.
But if your strategy still assumes that ranking equals visibility, it’s already behind.
Learn how to identify AI crawlers, review status codes, and spot access patterns in server logs.
A practical guide to improving visibility in AI-generated answers and search experiences.
How to format content so it is easier for answer engines and AI systems to interpret.