Micro-crawling: New way of indexing

In recent months, an unusual pattern has emerged on many websites: a drop in human traffic and, at the same time, a sudden increase in bot activity coming from unusual IP addresses, with unusual user agents and at unusual intervals. When I started analyzing the logs, a clear trend and pattern emerged that was not there months ago.

This article is an analysis of these changes, supported by real data, IP addresses, and behavior patterns, which I will publish in full in a PDF as evidence.

1. What happened after November?

Until November, traffic was stable. Then something happened that looks like a “switch flip”:

• human traffic from search engines went into free fall overnight
• bot traffic increased dramatically
• the server started to get overloaded
• the logs filled up with requests from AWS and other cloud IPs
• user agents became suspiciously similar to headless browsers

This was not the classic Googlebot. This was not Bingbot. This was not any known search agent.

This was something new.

2. AWS IPs and unusual user agents: a pattern that did not exist before

A huge number of requests from IPs such as the following began to appear in the logs:

34.245.xxx.xxx
52.214.xxx.xxx
54.171.xxx.xxx

These are typical AWS ranges used by:

• headless browsers
• scraping tools
• AI fetch systems
• micro-crawling for models
• test agents

The user agents were also suspicious:

• HeadlessChrome
• X11; Linux (no distribution)
• Chrome/139 (non-existent version)
• generic Linux UA without browser identification

These are not browsers used by people. These are automated tools.

3. Micro-crawling: 100 small visits instead of one large one

The classic Googlebot visits a page several times a day, in larger batches, with a clear user agent and from verified IP addresses.

What is happening now is different:

• small, fast, repetitive requests
• often only for images, CSS, or JS
• often only for part of the page
• intervals of 1–5 seconds
• different AWS IPs
• different headless agents

This is a textbook example of micro-crawling used by AI systems for real-time data refresh.

4. Why is this happening? AI needs fresh data

AI systems (Gemini, ChatGPT Search, Perplexity, etc.) need:

• real-time data
• granular data
• partial data
• structured data

Instead of one large crawl per week, we now see 100 small checks per day.

This is no longer classic indexing. This is real-time data capture.

5. Decline in human traffic: a logical consequence

When AI systems start responding directly in the search engine, the following happens:

• the user gets an answer without clicking
• AI uses your content, but you don’t get any visits
• traffic drops
• bot traffic increases
• the server gets overloaded

This is exactly what happened to me — and it’s exactly what’s happening to many others.

6. Evidence: IPs, patterns, and logs

The attached PDF contains:

• complete logs
• timestamps
• IPs (partially anonymized)
• user agents
• request frequencies
• examples of micro-crawling
• examples of AWS traffic spikes
• 7. What does this mean for authors?

It means:

• more bot traffic
• less human traffic
• more server load
• more costs
• less revenue
• less control over your own content

8. Conclusion: Micro-crawling is the new reality

If we combine all the data — unusual IPs, unusual user agents, real-time checks, traffic drops, growth in bot activity, and timing consistent with AI development — we get a clear picture: The web is increasingly being used as free fuel for search and AI systems, while authors bear the costs and lose traffic.