# Your robots.txt Is Now an AI Policy Document

Source: https://aicommercewatch.com/blog/2026-11-12-robots-txt-ai-policy-document.html

November 12, 2026

*Every AI surface reads your store through data you publish: a product feed, structured data, and the crawler rules in robots.txt. A blanket disallow blocks the shopping agents and the answer engines at once. Write the file per bot, per surface, and revisit it when a new assistant launches.*

## The file nobody rewrote

Most storefronts have a robots.txt that was written for Googlebot in 2019 and never touched again. It disallows the admin paths, the cart, the search results, and sometimes, by copy-paste, everything a crawler might want. In 2019 that was a search decision. In 2026 it is an AI decision, because the same file now governs GPTBot, ClaudeBot, PerplexityBot, and the shopping agents that read product data. A blanket disallow does not only keep a bot from indexing a page. It keeps the store out of the answers and the shopping surfaces that bot feeds, and the store never sees the absence, because absence leaves no log line in any dashboard.

The feed is the other half of the same decision. Every major AI shopping surface ingests a structured feed as its primary product source. OpenAI's feed specification requires nine fields: item id, title, description, URL, brand, seller name, image URL, availability, and price. A listing with a missing availability field gives the agent nothing to act on, however persuasive the description. A listing with three plain sentences and complete Offer markup passes. The feed and the robots file are the same job: publishing the store in a form machines can read, on purpose.

## How to write it on purpose

Start with the surfaces that matter. ChatGPT reads the OpenAI product feed or Shopify Catalog, and its search crawler is OAI-SearchBot. Google AI Mode and Gemini read the Merchant Center feed. Perplexity reads Shopify Catalog, its Merchant Program, and PerplexityBot. One master feed can serve all of them, with a validation pass per surface, because they all read the same core fields: title, price, availability, brand, images, identifiers. The differences are in how they take the data in, and in a few field spellings, like pre\_order in OpenAI's format against preorder in Google's.

Then write the robots file to match. Allow the crawlers that feed the surfaces you want to be on, and disallow the ones you do not, by name, with a comment saying why, so the next person who reads the file knows it was a decision. Check the file quarterly, because a new assistant launches every few months and each one brings a new bot name. A robots.txt that nobody owns is a policy written by whoever copied it last, and the store is living with the consequences either way. The same feeds drive [AI shopping assistants recommending your products](https://aicommercewatch.com/blog/2026-11-10-ai-shopping-assistants-product-feed-errors.md) and [AI product images getting scale wrong](https://aicommercewatch.com/blog/2026-11-10-ai-generated-product-images-scale-errors.md).

## Frequently asked questions

### Do AI shopping agents read my website or my feed?

The feed comes first. Every major AI shopping surface ingests a structured feed as its primary product source, and the crawlers index the pages as a second source. Markup helps engines confirm the page matches the feed, but the feed is the entry ticket.

### What fields does a product feed need?

OpenAI's specification requires nine: item id, title, description, URL, brand, seller name, image URL, availability, and price. Optional fields that carry weight include GTIN, MPN, color, size, material, shipping, and review counts.

### Can one feed serve ChatGPT, Google, and Perplexity?

Usually yes, with a validation pass for each surface. They all read the same core fields, and differ in how they take the data in and in a few field spellings. Shopify Catalog syncs Shopify stores to several of them automatically.

### What happens if I block GPTBot?

The site will not be shown in ChatGPT search answers, per OpenAI's own documentation. That is a real tradeoff, not a technical detail, and it should be decided deliberately.