The AI Crawler Bill: What GPTBot, ClaudeBot, and PerplexityBot Cost Your Storefront
AI crawlers like GPTBot, ClaudeBot, and PerplexityBot request your HTML directly, so they never load your analytics tag and never show up in GA4. Filter your server logs by their published user-agent tokens, verify the IPs against the vendors' published ranges, and group hits by URL and status code. That is the only honest count of AI traffic.
The traffic your analytics cannot see
Open your analytics and look at yesterday. Some of those visitors were not people, and the ones you cannot see at all were machines that never ran your tracking script. AI crawlers fetch pages on the server: they request the HTML, read it, and leave. No JavaScript runs, no tag fires, no session is counted. Your raw access logs or CDN logs are the record, and most stores have never looked at them. One log analysis guide measured 12,099 AI and crawler requests drawn from 288,566 log files over 48 days, and the point of the exercise was not the volume. It was that none of it appeared in any dashboard the store owned.
The names to filter for are published by the vendors themselves: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User. Group the hits by URL and status code, and two facts emerge. First, crawler hits are a precondition for being cited, not evidence of it: an assistant can crawl your product page a hundred times and never name your brand in an answer. Second, the pages that fail are invisible too: a product page that returns a 403 to GPTBot looks fine in every dashboard you own, and the assistant simply never cites it.
What the bill actually is
The cost is not bandwidth. A product page is kilobytes, and a crawl is cheap. The cost is three things that are harder to price. Server load during crawl bursts, when a new model trains or an index refreshes and thousands of requests arrive in an hour. Stale or blocked pages that never get cited, which is lost discovery you cannot measure because the absence of a citation leaves no trace. And the verification work: a user-agent string can be faked, so a count you report without verifying IPs against the vendors' published ranges is a count of strings, not of crawlers.
The decision most stores avoid is whether to let them in at all. Blocking GPTBot keeps your content out of ChatGPT's search answers, which OpenAI states directly. Blocking Google-Extended does not affect search inclusion or rankings, because it is a robots.txt control token, not a crawler with its own user agent. Those are different decisions with different consequences, and they should be made per bot, per surface, on purpose, not by a blanket disallow that someone added to robots.txt in 2023 and never revisited. The citation side of this is answer engines citing product pages, and the hallucination side is your chatbot quoting a price you never set.
Frequently asked questions
Why does Google Analytics not show AI bot traffic?
Analytics tools rely on a script running in a browser. A crawler requests the HTML and reads it without ever loading the script, so the visit is never counted. Server logs are the only record.
Which user-agent tokens should I filter for?
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, and Perplexity-User. Verify the IPs against each vendor's published ranges, because a user-agent string can be faked.
Does blocking a crawler hurt search rankings?
It depends on the crawler. Blocking GPTBot keeps the site out of ChatGPT search answers. Google-Extended is a robots.txt control token with no separate user agent, and Google says it does not affect inclusion in Search or act as a ranking signal.
How do I know if crawlers are being blocked?
Check status codes in the server logs, grouped by bot. A product page returning 403 to GPTBot looks fine in every dashboard, and the assistant simply never cites it.