Robots.txt for WordPress: A Safe Starter File, Line by Line

A safe robots.txt for WordPress is short. Block the admin area, keep admin-ajax.php open, optionally keep crawlers out of internal search results, and list your XML sitemap. Do not block /wp-content/, /wp-includes/, CSS, JavaScript or any page you want removed from search. Robots.txt controls crawling, not indexing, so it is the wrong tool for hiding pages.

This guide explains what WordPress already serves, walks through a starter file line by line, and shows how Google decides which rule wins. It also covers blocking AI crawlers, editing the file safely and checking it afterwards. We tested every rule against Google's open-source robots.txt parser, which Google describes as the production code Googlebot uses to decide which URLs it may fetch. Two of those tests change the advice you will often read elsewhere.

What robots.txt does and does not do

Robots.txt is a plain text file at the root of a host, such as https://example.com/robots.txt. Compliant crawlers read it before fetching other URLs and skip anything it disallows. Google's documentation describes its main purpose as managing crawler traffic so your server is not overloaded.

Table: What robots.txt can and cannot do

Goal	Does robots.txt work?	Use instead
Stop crawlers wasting requests on admin or search URLs	Yes	N/A
Keep a page out of Google's results	No. A blocked URL can still be indexed, without a description, if other pages link to it	A noindex robots meta tag or X-Robots-Tag header
Protect private files	No. The file is public and only polite crawlers obey it	Password protection or server rules
Remove a page that is already indexed	No. Blocking stops Google seeing a noindex tag	Allow crawling and add noindex, or delete the page (404/410)
Point crawlers to your sitemap	Yes, with a Sitemap: line	Also submit it in Search Console

Rules apply only to the exact host, protocol and port that serves the file. A subdomain such as shop.example.com needs its own file. Paths are case-sensitive, but field names such as Disallow are not.

<h2

Before you start

• Access. You need SFTP, your host's file manager, or an SEO plugin that can edit the file.
• Your sitemap URL. WordPress core serves /wp-sitemap.xml. Yoast SEO replaces it with /sitemap_index.xml. Open the URL in a browser and confirm it loads before you list it.
• The current file. Open https://yoursite.com/robots.txt and save a copy of what is there, so you can roll back.
• Staging and live sites. Check which site you are editing. A staging rule copied to production is the most common robots.txt disaster.

WordPress's virtual robots.txt

Most WordPress sites have no robots.txt file on disk. When a request for /robots.txt reaches WordPress, the core function do_robots() generates one. On a standard install the output is:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Plugins can add to this output through the robots_txt filter. Core's sitemap feature uses it to append a Sitemap: line, and SEO plugins may add their own lines.

Two details are worth knowing:

• A real file wins. If a physical robots.txt exists in the web root, the web server serves it directly and WordPress never runs. Plugin additions stop appearing, which surprises people who created a file once and forgot about it.
• "Discourage search engines" does not edit robots.txt. Since WordPress 5.3, the Settings → Reading checkbox adds a noindex robots meta tag to pages instead of a Disallow: / rule. If you are checking whether a site is hidden from search, look at the page source, not just robots.txt.

<h2

A safe starter file, line by line

This is the starter file we recommend for a typical WordPress site. Replace the sitemap URL with your own.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /*?s=
Disallow: /search/

Sitemap: https://example.com/wp-sitemap.xml

Table: What each line does

Line	Meaning	Why it is there
User-agent: *	Starts a group of rules for every crawler without a more specific group	One group keeps the file simple
Disallow: /wp-admin/	Blocks every URL that starts with /wp-admin/	The dashboard has nothing to rank and logged-out crawlers only see redirects to the login page
Allow: /wp-admin/admin-ajax.php	Reopens one file inside the blocked folder	Themes and plugins load front-end content through it; the longer, more specific rule wins
Disallow: /*?s=	Blocks any URL containing ?s=	WordPress search results can create endless thin URLs
Disallow: /search/	Blocks pretty search URLs such as /search/shoes/	Some themes and plugins rewrite search to this form
Sitemap: ...	Gives the absolute URL of your sitemap	Not tied to any group; you can list several

Notice the search rule. Many starter files, including the earlier version of this guide, use Disallow: /?s=. That rule only matches searches made from the home page. Our test below shows why the wildcard version is safer.

Worked example: testing the starter file

We built Google's open-source parser and checked sample URLs against both versions of the search rule with the Googlebot user agent. These results come from that test; your own URLs will differ.

Table: Parser results for sample WordPress URLs (Googlebot)

URL path	With Disallow: /?s=	With Disallow: /*?s=
/wp-admin/options.php	Blocked	Blocked
/wp-admin/admin-ajax.php	Allowed	Allowed
/?s=shoes	Blocked	Blocked
/page/2/?s=shoes	Allowed	Blocked
/blog/?s=x	Allowed	Blocked
/shop/?size=m	Allowed	Allowed
/WP-ADMIN/	Allowed	Allowed
/wp-content/uploads/a.jpg	Allowed	Allowed

Three lessons come out of this. First, /?s= misses paginated and section-level search URLs, so use /*?s=. Second, /shop/?size=m is not caught, because ?s= must appear exactly; the rule does not block every parameter that starts with "s". Third, paths are case-sensitive, so /WP-ADMIN/ is not matched by a lowercase rule. That does not matter on most WordPress servers, but it matters if you try to block a folder whose case varies.

If your site uses another search parameter (some shop and directory plugins do), add a rule for it, such as Disallow: /*&s= for a search parameter that appears after another one.

How Google decides which rule wins

Google's specification gives three rules that explain almost every surprise:

• Only one group applies to a crawler. Google picks the group with the most specific matching user agent and ignores the rest. A User-agent: Googlebot group replaces the * group for Googlebot; the two are not combined.
• The longest matching path wins. Inside a group, the rule with the longest matching path takes precedence, whatever order the lines appear in.
• Ties go to Allow. If an Allow and a Disallow rule match with equal length, the less restrictive rule applies.

Rules are prefix matches. * matches any run of characters and $ marks the end of the URL. This next test shows how that plays out:

Table: Pattern behaviour in Google's parser

Rule	URL path	Result
Disallow: /tag	/tagline-ideas/	Blocked (prefix match; probably not intended)
Disallow: /tag/ plus Allow: /tag/news/	/tag/news/	Allowed (longer rule wins)
Disallow: /*.pdf$	/file.pdf	Blocked
Disallow: /*.pdf$	/file.pdf?v=2	Allowed ($ requires the URL to end there)
Disallow: /*?replytocom=	/post/?replytocom=5	Blocked

The first row is the classic mistake. Always end folder rules with a slash unless you really mean "everything that starts with these letters".

Do not block these

• /wp-content/ and /wp-includes/. They hold your theme's CSS and JavaScript, your images and core scripts. Google renders pages like a browser, so blocking them can make pages look broken to Googlebot. In our test, Disallow: /wp-content/ blocked /wp-content/themes/x/style.css.
• Pages you want removed from search. Add noindex and leave them crawlable until they drop out. If you block them, Google cannot see the tag.
• Category, tag or author archives you are unsure about. If they are thin, noindex them through your SEO plugin instead; that keeps links on them followable.
• Your feeds and sitemaps. Blocking /feed/ rarely helps, and blocking the sitemap undoes the Sitemap: line.
• The whole site. Disallow: / is only for staging sites, and even there password protection is safer.

Optional: rules for AI crawlers

Some site owners add groups for AI crawlers. Google documents Google-Extended as a robots.txt product token that controls whether content Google crawls can be used to train future Gemini models and for grounding. It has no separate user agent string, and Google states that it does not affect inclusion or ranking in Google Search. Other companies publish their own tokens, such as GPTBot. Compliance is voluntary.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /*?s=

User-agent: GPTBot
User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/wp-sitemap.xml

Remember that groups are not merged. A crawler matched by the second group ignores everything in the * group. That is harmless here because the second group blocks everything, but if you write a partial group for one bot (for example, Disallow: /private/ only), that bot will no longer obey your admin or search rules unless you repeat them.

Decide this as a business question, not a technical default. Blocking AI crawlers does not remove content already collected, and it does not stop crawlers that ignore robots.txt.

Creating or editing the file

Option 1: a physical file

• Write the file in a plain text editor and save it as robots.txt in UTF-8.
• Upload it to the folder that contains wp-config.php and wp-load.php (the web root).
• Open /robots.txt in a private browser window to confirm the new content appears.

Option 2: your SEO plugin

Yoast SEO includes a file editor tool that can create robots.txt when the web root is writable and edit it when the file is writable. If the option is missing, your host may not allow file writes; use SFTP instead.

Option 3: the robots_txt filter (developers)

If you prefer to keep WordPress generating the file, you can append rules in a small plugin or must-use plugin. This only works when there is no physical file.

<?php
// wp-content/mu-plugins/robots-rules.php
add_filter( 'robots_txt', function ( $output, $public ) {
    if ( $public ) {
        $output .= "Disallow: /*?s=n";
        $output .= "Disallow: /search/n";
    }
    return $output;
}, 10, 2 );

The $public argument is false when "Discourage search engines" is ticked, so the extra rules are skipped on sites that are not meant to be indexed.

Whichever option you choose, you can draft rules with the robots.txt generator, which has a WordPress preset and optional AI crawler groups.

Verify the file after every change

• Check the response. /robots.txt should return status 200 as plain text. Check it with the HTTP status checker. Google treats most 4xx responses as "no restrictions" but stops crawling for a period after 5xx errors, so a broken server response matters more than a broken rule.
• Test important URLs. Paste the file into the robots.txt tester and test your home page, a post, a product or category page, a CSS file and an image. All should be allowed. Then test an admin URL and a search URL, which should be blocked.
• Confirm the sitemap. Run your sitemap through the sitemap URL extractor and spot-check that listed URLs are not blocked by your new rules.
• Check Search Console. The robots.txt report under Settings shows the version Google last fetched, its fetch status and any lines Google could not parse. After an urgent fix, use "Request a recrawl" from the report.
• Allow for caching. Google generally caches robots.txt for up to 24 hours, so changes are not instant.

Common mistakes

• Leaving Disallow: / from staging on a live site.
• Using robots.txt to hide pages, then wondering why they appear in results with no description.
• Blocking /wp-content/ or /wp-includes/ to "save crawl budget".
• Writing Disallow: /tag without a trailing slash and blocking unrelated pages that start with "tag".
• Adding a bot-specific group and forgetting that it replaces the * rules for that bot.
• Putting Noindex: or Crawl-delay: lines in the file and expecting Google to obey them. Google supports only user-agent, allow, disallow and sitemap.
• Using a relative sitemap path. The Sitemap: line needs a full URL including https://.
• Uploading a physical file and then wondering why plugin-generated lines disappeared.

Troubleshooting

Table: Robots.txt problems and fixes

Symptom	Likely cause	Fix
Search Console: "Indexed, though blocked by robots.txt"	Google found links to a blocked URL and indexed it without crawling	If the page should be hidden, unblock it and add noindex. If it should rank, remove the Disallow rule
Search Console: "Blocked by robots.txt" on pages you want indexed	A rule is broader than intended (often a missing trailing slash or wildcard)	Test the URL in a tester, find the matched rule and narrow it
Your edits do not appear at /robots.txt	A caching plugin, CDN or a second physical file is serving an old copy	Purge caches and check the web root for the file
Plugin-added sitemap line has vanished	A physical file now overrides the virtual one	Add the Sitemap: line to the physical file, or delete the file
Site-wide drop in crawling	/robots.txt returning 5xx, or Disallow: / left in place	Fix the server response first, then the rules, and request a recrawl
Pages look unstyled in Google's URL Inspection screenshot	CSS or JavaScript blocked	Remove rules that block /wp-content/, /wp-includes/ or file types such as .css and .js

If a page is still hidden after you fix robots.txt, check its meta robots tag and X-Robots-Tag header with the noindex checker. Robots.txt and noindex problems often look alike in reports.

Final checklist

• The file returns 200 and plain text at the root of each host you use.
• Admin is blocked and admin-ajax.php is allowed.
• Search URLs are blocked with a wildcard rule.
• No rule blocks /wp-content/, /wp-includes/, CSS, JavaScript or images.
• Folder rules end with a slash.
• Any bot-specific group repeats the rules you still want that bot to follow.
• The sitemap line uses an absolute URL that loads.
• Pages you want hidden use noindex, not Disallow.
• Search Console's robots.txt report shows the new version with no issues.

Frequently asked questions

Does WordPress create a robots.txt file automatically?

It generates a virtual one when no physical file exists. The default output blocks /wp-admin/ and allows /wp-admin/admin-ajax.php, and plugins can add lines such as a sitemap reference.

Where do I put robots.txt in WordPress?

In the web root, the same folder as wp-config.php in a standard install, so it loads at https://yoursite.com/robots.txt. Files in subfolders are ignored.

Will robots.txt stop a page appearing in Google?

No. Google can still index a blocked URL if other pages link to it, usually without a description. Use a noindex tag or header and leave the page crawlable.

Should I block wp-content or wp-includes?

No. Those folders contain the CSS, JavaScript and images Google needs to render your pages. Blocking them can make pages appear broken to Googlebot.

Should I block WordPress search results?

It is usually sensible, because search URLs can multiply endlessly. Use Disallow: /*?s= so paginated and section-level searches are covered, not only searches from the home page.

Does Google support Crawl-delay?

No. Google ignores crawl-delay and other unsupported fields. It supports only user-agent, allow, disallow and sitemap. Other search engines may treat it differently.

Does blocking Google-Extended affect my rankings?

Google states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls use of content for Gemini training and grounding.

How long does Google take to see robots.txt changes?

Google generally caches the file for up to 24 hours. You can request a recrawl from Search Console's robots.txt report after urgent fixes, but the timing is not guaranteed.

Is the "Discourage search engines" setting the same as Disallow: /?

No. Since WordPress 5.3 it adds a noindex robots meta tag to pages and leaves robots.txt rules alone. That is better for keeping a site out of results, but password protection is safer for staging.

Do I need a robots.txt file at all?

No. If there is no file, crawlers assume they may crawl everything. On WordPress the virtual file already exists, so you only need a physical file if you want custom rules or your plugin cannot add them.

Next step

Create or update your file with the robots.txt generator, then test your key URLs in the robots.txt tester before you upload it. If you also manage server rules, our guide to the default WordPress .htaccess file covers safe additions.