Quick answers
Direct answers to the questions readers ask most.
What are AI crawlers and why are they on my store?
Automated readers from AI companies collecting training data and grounding shopping answers - GPTBot, ClaudeBot, PerplexityBot and others. They read your catalog either to build models or to power AI product recommendations.
Should I block AI crawlers from my ecommerce site?
Not blanket. Answer engines that cite and link can send qualified shoppers; bulk harvesters mostly take. Classify first, then allow, throttle, or block per class - robots.txt alone is intent, not enforcement.
What is GPTBot and does it send traffic?
GPTBot is OpenAI's web crawler for collecting training data. It reads pages but does not itself send shoppers back; answer features and agentic browsing are separate systems with different behavior.
AI assistants completing purchases on behalf of users - researching, deciding, and checking out autonomously. It turns 'bots' from a purely hostile category into one that can carry revenue.
How do I see which AI crawlers visit my store?
Server or CDN logs filtered by AI user-agents give a raw view; dedicated edge detection products classify crawlers by operator and behavior, including ones that disguise themselves.