AI Commerce WatchTracking the agents that shop

Sep 15, 2026 - AI Commerce Watch

The Merchant's Guide to AI Crawlers: Who Is Reading Your Catalog and Why

Two years ago, the crawlers hitting your store were search engines and scrapers. Today a third category is growing fast: AI crawlers collecting training data and grounding shopping answers. Most merchants have no policy on them - because most have no visibility into them.

The three jobs AI crawlers do

Training crawlers (like GPTBot and ClaudeBot) read pages to build models. They send no traffic back today; the value exchange is debatable. Answer-engine crawlers (PerplexityBot and similar) read pages to ground live answers - when someone asks an AI assistant where to buy a product, these are the bots that decided what it knows. Agentic crawlers act for a user in real time: checking prices, comparing specs, and in some cases initiating checkout. This last group is small today and strategically enormous.

Why this is an ecommerce story, not a tech story

Product discovery is migrating into AI answers. If your catalog is not readable by answer engines, you are invisible in a growing discovery channel. If it is readable by everyone indiscriminately, your pricing and merchandising logic are free for competitors and arbitrageurs. Both errors are expensive; the difference between them is a policy.

A sane crawler policy in three steps

First, get visibility: which AI user-agents hit your storefront, at what rate, and what they read. robots.txt directives are statements of intent, not measurements - actual behavior needs observation at the server or edge. Second, classify: answer engines that cite and link are a channel; bulk harvesters are a cost. Third, enforce: allow, throttle, or block per class. Purpose-built tools now exist for exactly this visibility layer.

The takeaway

The AI crawler question is not 'block or allow.' It is 'which bots, doing what, at what cost, and on whose terms.' Merchants who can answer that will set the terms; everyone else will accept them.

Frequently asked questions

What are AI crawlers doing on a storefront?

Reading product information to answer shopper questions, build comparisons, and increasingly to shop on a customers behalf. They consume titles, descriptions, prices, stock, and policies, which makes the accuracy of that data a sales issue, not a technical curiosity.

Should a store block AI crawlers?

Blocking is a choice with a cost: stores that block shopping agents and answer engines disappear from the recommendations those systems make. Most stores are better served by allowing reputable crawlers and controlling access through their robots rules deliberately.

How can a store check what crawlers see?

Fetch key product pages as a crawler would, check the structured data, and read the plain text version of the page. If prices, variants, or stock are missing or stale in that view, that is what AI systems repeat about the store.

What data matters most to AI shopping systems?

Current price and availability, clear variant and sizing information, delivery and returns terms, and consistent product identifiers. Agents recommend what they can verify, and they skip listings whose data contradicts itself.