How robots.txt rules are read
A robots.txt file is a set of groups. Each group starts with one or more User-agent lines and lists Allow and Disallow rules for those crawlers. Google follows three rules when it reads the file:
- One group applies. A crawler obeys the group that names it most specifically, and only falls back to
User-agent: *when no group names it. Rules from different groups are not combined. - The longest matching path wins.
Allow: /admin/publicbeatsDisallow: /adminfor a URL under /admin/public. - Allow wins a tie. When two matching rules are the same length, the page is allowed.
Two wildcards are supported: * matches any run of characters, and a trailing $ anchors the end of the URL, so Disallow: /*.pdf$ blocks PDF files but not a page called /pdf-guide.
Mistakes that quietly hurt your rankings
- A leftover Disallow: / from a staging site, which blocks the whole site. It is the single most damaging line a launch can ship with.
- Blocking CSS and JavaScript. Google renders pages like a browser. If it cannot load your scripts, it may not see your content.
- Using robots.txt to hide pages. Blocked pages can still be indexed from links elsewhere, and Google can no longer see a noindex tag on them.
- Rules before any User-agent line. Crawlers ignore them.
- Relying on directives Google ignores. Google does not support
crawl-delay,host, ornoindexin robots.txt.
A sensible robots.txt for a new product
User-agent: * Disallow: /dashboard Disallow: /api/ Sitemap: https://yoursite.com/sitemap.xml
Block the app areas that have no search value, allow everything else, and point to your sitemap. Most early-stage sites need nothing more.
AI crawlers and your robots.txt
AI companies use separate user agents for separate jobs, so one blanket rule rarely does what you intend:
| User agent | Used for |
|---|---|
| GPTBot | Collecting content to train OpenAI models |
| OAI-SearchBot | Showing your pages in ChatGPT search results |
| Google-Extended | Gemini training and grounding. It does not affect Google Search. |
| ClaudeBot, PerplexityBot, CCBot | Anthropic, Perplexity, and Common Crawl crawling |
If you want buyers to find you through AI answers, think twice before blocking the search-focused agents. Our guide to tracking your visibility in AI search covers how to measure whether those answers mention you, and the sitemap validator checks the file your Sitemap line points to.
Frequently asked questions
- How do I test my robots.txt?
- Enter your domain above. The validator fetches /robots.txt, reports syntax problems and risky rules, and gives you a tester that shows whether a path is allowed for Googlebot, Bingbot, or an AI crawler, and which line decided it.
- Does Disallow stop a page from appearing in Google?
- No. Disallow stops crawling, not indexing. A blocked page can still appear in results if other sites link to it. To keep a page out of search, allow crawling and add a noindex robots meta tag.
- What happens if I have no robots.txt?
- Crawlers treat a missing file (a 404) as permission to crawl everything. That is fine for most small sites, though a robots.txt is still the easiest place to point crawlers at your sitemap.
- Which rule wins when Allow and Disallow both match?
- For Google, the most specific rule wins, meaning the longest matching path. If an Allow and a Disallow are equally long, Allow wins.
- Should I block AI crawlers like GPTBot?
- It depends on whether you want your content used by AI products. Blocking GPTBot or Google-Extended limits training use, but crawlers such as OAI-SearchBot decide whether you can appear in AI search answers, so block deliberately rather than all at once.