Most Shopify stores should let AI search and user-fetch bots in, and make a deliberate call on AI training bots. Shopify's default robots.txt already allows all of them to read your product and collection pages. This guide sorts the bots by job, shows the exact lines to change in robots.txt.liquid, and explains what llms.txt is worth in 2026.
If you want the wider picture first (product data, checkout channels, Shopify's agent settings), start with Is your store ready for ChatGPT and AI shopping agents?. This post goes one level deeper on crawler access.
Which AI crawlers should a Shopify store allow?
Allow every AI search bot and every user-fetch bot: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot and Claude-User. They decide whether assistants can find, cite and link your products. Training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) are a separate business decision that doesn't change whether you get cited in search answers.
The mistake we see most is treating "AI bots" as one group. A store that blocks everything with "AI" in the name to protect its content also removes itself from ChatGPT search and Perplexity answers, which is usually the opposite of what the merchant wanted.
What is the difference between training, search and user-fetch bots?
Training bots collect pages that may be used to train future AI models. Search bots build the index an assistant searches when it answers a question. User-fetch bots load a specific page because a person asked the assistant to look at it. Only the last two affect whether a shopper sees your product in an answer today.
Here is how each vendor describes its own bots.
OpenAI
OpenAI's crawler documentation lists three you need to know. OAI-SearchBot surfaces sites in ChatGPT search, and "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers." GPTBot crawls content that may be used for training; disallowing it "indicates a site's content should not be used in training generative AI foundation models." ChatGPT-User handles actions a user starts in ChatGPT, and because those are user-initiated, OpenAI says "robots.txt rules may not apply."
Perplexity
Perplexity's bot documentation says PerplexityBot is "designed to surface and link websites in search results on Perplexity" and is not used to crawl content for AI foundation models. Perplexity-User visits a page when someone asks Perplexity a question, and "since a user requested the fetch, this fetcher generally ignores robots.txt rules."
Anthropic
Anthropic's crawler article names three bots. ClaudeBot collects content that could contribute to model training. Claude-SearchBot indexes content for search, and blocking it "may reduce your site's visibility and accuracy in user search results." Claude-User retrieves pages when a person asks Claude a question; blocking it "may reduce your site's visibility for user-directed web search." Anthropic says its bots honor robots.txt and warns that blocking its IP addresses instead may not work as an opt-out, because it stops them from reading your robots.txt.
Google-Extended is not a crawler at all. Google says it "doesn't have a separate HTTP request user agent string"; it is a robots.txt token that controls whether content Googlebot already crawled is used to train Gemini models and for grounding in Gemini Apps and Vertex AI. Google is explicit that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." AI Overviews and AI Mode run on the normal Googlebot index.
Apple
Apple's Applebot page works the same way. Applebot powers search in Spotlight, Siri and Safari. Applebot-Extended "does not crawl webpages"; disallowing it opts your content out of training Apple's foundation models, and your content "will remain discoverable through Spotlight, Siri, and Safari."
Common Crawl
CCBot builds Common Crawl's free, public archive of the web, which anyone can download. It respects robots.txt. It powers no assistant directly, so blocking it only keeps your future pages out of that archive.
What does Shopify's default robots.txt allow?
Shopify's default robots.txt allows every crawler, AI bots included, to read public product, collection, page and blog URLs. It blocks admin, cart, checkout and account paths, plus sorted and filtered collection URLs that create duplicate pages. It has no rules aimed at AI bots, so if you never edited it, they are already in.
Shopify's robots.txt documentation describes those defaults. On the Shopify storefronts we checked in October 2026, the generated file now also opens with comments pointing agents to the store's agents.md file and its Universal Commerce Protocol (UCP) endpoints, which Shopify's changelog confirms every storefront advertises at /.well-known/ucp.
So for most stores the job is an audit, not a rewrite. Open yourstore.com/robots.txt and look for any group that names an AI bot. If an app, a past agency or a copied snippet added blanket blocks, that is the thing to fix.
How do you edit robots.txt on Shopify?
You edit robots.txt by adding a robots.txt.liquid template to your theme. In the Shopify admin, go to Online Store, open the theme menu, choose Edit code, add a new template, and pick robots. Shopify generates the default rules as Liquid; you add your own groups around them and save.
Two warnings from Shopify's help page are worth repeating. It is an unsupported customization, so "Shopify Support can't help with edits to the robots.txt.liquid file." And "incorrect use of the feature can result in loss of all traffic." Delete the template and Shopify falls back to its default file.
Shopify's developer docs for the template recommend keeping the Liquid loop over robots.default_groups that outputs the defaults, then adding custom groups as plain text below it. Rewriting the whole file by hand means you stop receiving Shopify's future updates to it.
What robots.txt rules should you add for AI bots?
Usually none. If you want to stay in AI answers but keep your content out of model training, add one group below Shopify's default loop that disallows only the training tokens. Leave the search and user-fetch bots under Shopify's defaults so they keep reading products and collections.
A training-only opt-out group looks like this, one line each, in this order:
- User-agent: GPTBot
- User-agent: ClaudeBot
- User-agent: Applebot-Extended
- User-agent: CCBot
- Disallow: /
Think twice before adding Google-Extended to that group. Google says the token also covers grounding in Gemini Apps, so blocking it can limit how Gemini uses your pages when it answers shoppers. Many brands decide product pages are marketing and leave every bot open. That is a reasonable default for a store whose goal is to be recommended.
Never add OAI-SearchBot, PerplexityBot or Claude-SearchBot to a Disallow group unless you mean to leave those assistants' search answers. Don't use robots.txt to fight ChatGPT-User or Perplexity-User either: both vendors say those fetchers may not follow it.
Finally, check what sits in front of your store. robots.txt only matters if the bot can reach the file. A bot-management rule or IP block that challenges these crawlers overrides everything above, and Anthropic specifically warns that IP blocking interferes with reading robots.txt.
What is llms.txt, and does it help?
llms.txt is a proposed plain-markdown file at the root of a site that summarizes it for language models and links to the pages that matter. Jeremy Howard published the proposal in September 2024. It is meant for AI tools at answer time, not for training. No major AI search engine has said it ranks or cites sites based on it.
Google is the clearest. Its guide to AI features in Search says "you don't need to create new machine readable files, AI text files, or markup to appear in these features," and that there are no additional requirements for AI Overviews or AI Mode beyond being indexed and eligible for a snippet.
Shopify has also moved here. The storefronts we checked already serve a Shopify-generated llms.txt that tells agents how to shop the store through Shop and UCP. Visit yourstore.com/llms.txt to see yours before you write anything.
Our view: treat llms.txt as cheap housekeeping, never as a growth lever. The work that changes AI visibility is crawler access, product facts in the raw HTML, and structured data that matches the page. See our guides to product pages AI shopping assistants recommend and Shopify product structured data in 2026.
How do you check what AI bots can reach on your store?
Run your domain through the free DYODE store scan. It reads your live robots.txt and reports whether GPTBot, PerplexityBot, ClaudeBot and Google-Extended can reach your home page and a sampled product page, then checks Product and Organization schema and whether the product title and price are in the raw HTML.
The scan doesn't read llms.txt or test OAI-SearchBot, so check those two by hand: open the file in a browser and confirm no group disallows OAI-SearchBot. Once bots can get in, track ChatGPT, Perplexity and Gemini traffic in GA4 to see whether those visits turn into sales.
If the scan flags a block you didn't put there, or you want robots.txt.liquid, schema and PDP content fixed in one pass, that is the core of DYODE's AEO and GEO optimization work. DYODE is a Shopify Plus partner in Orange County. Tell us about your store and we'll look at your crawler setup with you.
About DYODE. Founded in 2016 by former heads of E-Commerce at PacSun and Tilly’s. Shopify Premier Partner.
Add DYODE as a preferred source on Google
Google shows you more from the sources you pick.
