How to add your sitemap to robots.txt in WordPress

Step by step

1. Look at the live file

Open https://example.com/robots.txt in a browser. Look for a line that begins with Sitemap: followed by a full URL. A typical default file from WordPress reads:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

If the line is there and the URL opens a sitemap, you are finished.

2. Work out which robots.txt you have

WordPress generates robots.txt on request when no real file exists. To tell which you have, look in the site root (the folder containing wp-config.php) over FTP or your host's file manager. If a file named robots.txt is there, the web server sends it directly and WordPress, plugins and their sitemap lines never get involved. Hydrogen SEO's Robots.txt Editor shows a warning and locks itself in that case.

3. With a virtual file: let the plugin handle it

With Hydrogen SEO active, the sitemap line is added automatically and points at the plugin's sitemap index. It is kept when you edit rules in Hydrogen SEO → Robots.txt Editor, and View Robots.txt opens the live file so you can confirm. The plugin switches the core sitemap off, so you will see one Sitemap: line for /sitemap.xml and none for /wp-sitemap.xml.

4. With a physical file: add the line or remove the file

You have two choices. The simpler one is to delete or rename the physical file so WordPress serves the virtual one again, and move any custom rules into the plugin's editor. If the file has to stay, edit it and add the line yourself:

Sitemap: https://example.com/sitemap.xml

Use the address of your sitemap index, not each child sitemap. Save the file with plain UTF-8 encoding.

5. Follow the format rules

• Full URL. The value must be absolute, with https:// and the hostname. A path such as /sitemap.xml is not valid.
• Placement. The line is independent of User-agent groups, so it can go anywhere in the file. The end is conventional.
• Several lines are allowed. List more than one sitemap by repeating the line.
• Match the canonical host. If your site is https://www.example.com, the sitemap URL should use that exact scheme and host.

6. Check when the line is missing from a virtual file

Core only adds its sitemap line when the site is public. If Settings → Reading → Discourage search engines from indexing this site is ticked, the core sitemap is disabled and no line is printed. Untick it on a live site. The line is also absent when something disables the core sitemap with the wp_sitemaps_enabled filter without providing a replacement.

7. Test and move on

Open the URL from the Sitemap: line and confirm it returns XML with a 200 status:

curl -sI https://example.com/sitemap.xml | head -1

Then paste your robots.txt into the robots.txt tester to make sure no rule blocks the sitemap URL or the pages it lists.

Is the robots.txt line enough

For discovery by crawlers, yes. For reporting, no. Submitting the sitemap in Google Search Console and Bing Webmaster Tools shows you when it was last read, how many URLs were found and whether it has errors, none of which robots.txt can tell you. Do both; see submit a sitemap to Google.

Adding the line in code

Without an SEO plugin, when your sitemap comes from somewhere other than core, append to the virtual file with the robots_txt filter:

add_filter( 'robots_txt', function ( $output, $public ) {
    if ( $public ) {
        $output .= "\nSitemap: " . home_url( '/my-sitemap.xml' ) . "\n";
    }
    return $output;
}, 10, 2 );

The second argument is false when the site is set to discourage search engines, which is why the example checks it.