The Merchant's Guide to AI Crawlers: Who Is Reading Your Catalog and Why
Two years ago, the crawlers hitting your store were search engines and scrapers. Today a third category is growing fast: AI crawlers collecting training data and grounding shopping answers. Most merchants have no policy on them - because most have no visibility into them.
The three jobs AI crawlers do
Training crawlers (like GPTBot and ClaudeBot) read pages to build models. They send no traffic back today; the value exchange is debatable. Answer-engine crawlers (PerplexityBot and similar) read pages to ground live answers - when someone asks an AI assistant where to buy a product, these are the bots that decided what it knows. Agentic crawlers act for a user in real time: checking prices, comparing specs, and in some cases initiating checkout. This last group is small today and strategically enormous.
Why this is an ecommerce story, not a tech story
Product discovery is migrating into AI answers. If your catalog is not readable by answer engines, you are invisible in a growing discovery channel. If it is readable by everyone indiscriminately, your pricing and merchandising logic are free for competitors and arbitrageurs. Both errors are expensive; the difference between them is a policy.
A sane crawler policy in three steps
First, get visibility: which AI user-agents hit your storefront, at what rate, and what they read. robots.txt directives are statements of intent, not measurements - actual behavior needs observation at the server or edge. Second, classify: answer engines that cite and link are a channel; bulk harvesters are a cost. Third, enforce: allow, throttle, or block per class. Purpose-built tools now exist for exactly this visibility layer.
The takeaway
The AI crawler question is not 'block or allow.' It is 'which bots, doing what, at what cost, and on whose terms.' Merchants who can answer that will set the terms; everyone else will accept them.