Skip to content
My Blog

Search WPDesignVault

Try "image", "json", "schema" or "contrast". Use arrow keys to move through results.
SEO Tools

Robots.txt Generator

Create a clean robots.txt file with sitemap references, safe WordPress defaults and optional rules for AI crawlers, then download it.

Runs in your browser. Nothing you enter is uploaded. Last updated
Opt out of these AI crawlers (optional)

    How to use

    1. Choose whether crawlers are allowed by default.
    2. Add your sitemap URL and any paths you want to keep crawlers out of.
    3. Optionally tick the WordPress preset or AI crawlers you want to opt out of.
    4. Download robots.txt and upload it to the root of your domain.

    What this tool does

    The generator writes standard robots.txt directives (User-agent, Allow, Disallow, Sitemap) and checks that each path starts with a slash. It does not upload anything or connect to your site.

    Why it matters

    robots.txt tells well-behaved crawlers which parts of a site to skip, which saves crawl effort on pages that have no value in search. It is not a security tool: blocked URLs can still be visited, and a disallowed page can still appear in results if other sites link to it. Use a noindex tag to keep a page out of results.

    Tips

    • Never disallow your CSS and JavaScript files; search engines need them to render pages.
    • WordPress already serves a virtual robots.txt; a physical file in the site root replaces it.
    • Crawl-delay is ignored by Google.

    Frequently asked questions

    Where do I put robots.txt?

    In the root of the host, so it is reachable at https://yourdomain.com/robots.txt. Each subdomain needs its own file.

    Does blocking AI crawlers work?

    Crawlers that respect robots.txt will follow the rules for their user-agent token. It cannot stop crawlers that ignore the file.

    Related tools

    WordPress .htaccess Generator

    Generate a WordPress .htaccess file with the default rewrite rules and optional HTTPS and www redirects, file protection,…

    301 Redirect Generator

    Paste old and new URLs to generate exact-match redirect rules for Apache (.htaccess), Nginx or a WordPress snippet.…

    URL Slug Generator

    Convert titles into short, lowercase, SEO-friendly URL slugs. Removes accents and symbols, with optional stop-word removal. Works on…

    SERP Snippet Preview

    Preview a page title, URL and meta description as a search result on desktop and mobile.

    Meta Description Checker

    Check meta description length in characters and estimated pixels for desktop and mobile results, with a live preview.

    Meta Title Checker

    Check your page title length in characters and estimated pixels, and preview how it may look in search…