AI Commerce WatchTracking the agents that shop

Answered by AI Commerce Watch

Should I block AI crawlers from my ecommerce site?

Not blanket. Answer engines that cite and link can send qualified shoppers; bulk harvesters mostly take. Classify first, then allow, throttle, or block per class - robots.txt alone is intent, not enforcement.

A blanket block trades away a growing discovery channel; a blanket allow gives away catalog and pricing intelligence to anyone, including competitors. The working approach: measure which AI user-agents actually visit, classify them by behavior and value, then enforce per class.

robots.txt entries (like disallowing GPTBot) express preference, but compliance is voluntary and enforcement requires detection at the server or edge. Start with visibility, then write the policy the data supports.