Key numbers

Of 132,874 AI-bot requests logged across 19 of our sites between 5 August and 11 October 2026, 39,950 (30.1%) requested scanner-style paths such as /.env and /.git/config.
In the last 30 days (11 September to 11 October 2026), 28.4% of 79,577 AI-bot requests across 19 sites were scanner-style, down slightly from 30.1% over the full window.
In the "user-asked" tier (ChatGPT-User, Claude-User and similar), 49.0% of 22,135 requests were scanner-style, against 34.9% for retrieval bots and 23.1% for training bots (5 August to 11 October 2026, 19 sites).
Claude-User (60.5%) and Claude-SearchBot (57.7%) were among the bots with the highest scanner-style share in the last 30 days, while ClaudeBot (11.0%) and Amazonbot (9.7%) had the lowest of the larger bots (19 sites).
After removing scanner-style requests, the last 30 days of 56,970 AI-bot requests split into 66.6% training, 20.8% retrieval and 12.6% user-asked (19 sites).
The scanner-style requests came from 33 countries and 483 distinct paths, and the share ranged from 10.4% to 49.8% across the 7 sites with more than 9,000 AI-bot requests each (5 August to 11 October 2026).

Method

Data source

Each of our sites runs a small edge worker that records a request when the user agent matches a known AI bot. Each row holds the time, the site, the bot name, the path asked for, the country and a tier. We use the three AI tiers: training (for example ClaudeBot, GPTBot, Bytespider), retrieval (for example OAI-SearchBot, PerplexityBot) and user-asked (for example ChatGPT-User, Claude-User). Ordinary search-engine crawlers such as Googlebot and Bingbot are logged in a separate tier and are not part of this study.

Window

The full window runs from 5 August 2026 to 11 October 2026. The recent window is the 30 days to 11 October 2026 (from 11 September). Counts were taken on 11 October 2026.

How a request is counted as scanner-style

A request is scanner-style if its path matches a fixed list of patterns that no reader would ask for: .env, @fs, .git, wp-admin, wp-login, /vendor/, phpinfo, .aws, credentials, id_rsa, .ssh, /config.json, /backup, /fetch, /proxy, /graphql, /actuator, id_ed25519, terraform.tf, application.yml, .netrc and secrets.yaml. This is the same list our own analytics uses. It was built by looking at what scanners asked for, so it is a floor. A scanner that asks for a path we have not listed counts as a normal read.

What was removed first

Two kinds of row were removed before any counting. The first is requests that our own audit tools made to our own sites while wearing crawler user agents (2,104 AI-tier rows carry a marker saying so). The second is every row from Finland, because our audit tools run from a server in Finland and older rows have no marker. In the AI tiers, 27,540 rows from Finland were set aside this way, and only 0.1% of them had scanner-style paths, which supports treating them as our own traffic. The cost is that any real Finnish crawler traffic is lost too.

Bot names

Bot names are lower-cased before grouping, so "Meta-ExternalAgent" and "meta-externalagent" count as one bot. The per-bot tables only show bots with at least 100 requests in the window.

Trend

We counted by calendar month and by week. Because sites were added over time (7 sites in August, 19 by October), we also repeated the monthly count on a fixed set of sites that were already logging by 20 August.

Data

The aggregate tables are in the CSV linked from this page. It has counts only, with no IP addresses and no individual rows.

Findings

The headline

Across the full window, 30.1% of AI-bot requests on our sites were scanner-style. In the last 30 days it was 28.4%. We cannot prove each of these came from a scanner. We can say that a request for /.env is not a reader looking at a page.

The user-asked tier is the worst affected

The share of scanner-style requests rises as the bot type gets closer to "a person asked":

Training: 23.1% of 80,751 requests
Retrieval: 34.9% of 29,988 requests
User-asked: 49.0% of 22,135 requests

The user-asked tier is the one most often used as a sign that people are asking assistants about a site. Counted raw, about half of it is not that.

It differs a lot by bot

In the last 30 days, the bots with among the highest scanner-style shares included Claude-User (60.5%), Claude-SearchBot (57.7%), cohere-ai (56.8%), Perplexity-User (55.5%) and CCBot (54.2%). The lowest among the larger bots were Amazonbot (9.7%), ClaudeBot (11.0%), meta-externalagent (21.0%) and GPTBot (22.4%). A high share does not mean the vendor is doing this. It means that the name is being typed by someone else, which anyone can do.

What the clean traffic looks like

With scanner-style requests removed, the last 30 days hold 56,970 AI-bot requests. 66.6% are training bots, 20.8% are retrieval bots and 12.6% are user-asked. Even after cleaning, user-asked requests are the smallest group.

By week, the scanner-style share ran from 4.1% (week of 31 August) to 49.7% (week of 10 August). It comes in bursts, not as a steady background. Over the months on a fixed set of 7 sites it fell from 37.6% in August to 32.5% in September and 30.4% in October so far (11 days). That is a gentle fall with large swings inside it, so we would not call it a trend yet.

It depends on the site

On our 7 busiest sites the scanner-style share ran from 10.4% to 49.8%. Some smaller, newer sites showed none at all. We do not know why. One possibility is that the scanners work from older lists of targets and have not found the newer sites yet.

Two small things that changed

Amzn-SearchBot made only 64 requests in the last 30 days, against 2,944 over the full window, and most of those were scanner-style. Applebot (training tier) made none in the last 30 days. We did not look into why.

What this does not show

It does not show what any AI model does with pages it reads. A request is not a citation, a summary or an answer.
It does not measure how often our pages are cited or recommended by assistants. For that you need referral clicks or direct testing of the assistants, which this study did not do.
It covers only our own 19 sites, which are small and mostly new. Bigger or older sites will see different mixes.
It cannot tell a real bot from a forged one by user agent alone. We only separate them by the path asked for, so scanner-style traffic is a floor and some forged traffic is still counted as real.
It does not identify who is behind the scanner-style requests. The log has no IP addresses, so we cannot group them by network or check them against published bot IP ranges.
The counts come from our own edge logs. Cloudflare may sample some of what feeds them, so treat small numbers as presence or absence, not exact volume.
It does not include Finnish traffic, which was set aside because our own audit tools run from Finland. Real crawler traffic from Finland is missing.
Some of the first weeks of data were collected before a fix to how low-volume bots were recorded on 4 August 2026. The window starts on 5 August, but earlier summary tables could be missing low-volume bots. This study uses the row-level log and is not affected, but the log began with 7 sites, so early weeks are less broad.

Data and citation

Data: https://noctemlabs.co.uk/research/ai-crawler-traffic/data.csv, licensed CC BY 4.0.

Stovell, J. (2026). How much "AI crawler" traffic is actually AI? Noctem Labs. https://noctemlabs.co.uk/research/how-much-ai-crawler-traffic-is-ai/

How much "AI crawler" traffic is actually AI?