Crawl Cleanup

WordPress adds a number of tags, headers and feed addresses to every site, whether or not the site uses them: a shortlink for each post, links that announce the REST API and remote publishing, a tag stating the WordPress version, a script for emoji in old browsers, and a comments feed for every post. From version 0.0.31, Crawl cleanup gives you fourteen switches to remove the ones your site does not need. Every switch is off by default, and nothing on your site changes until you turn one on. It is free and needs no account.

Where to find it

Open Hydrogen SEO → General and expand Crawl cleanup. Its subtitle reads "Remove tags, headers and feeds WordPress adds that your site may not need. All off by default". The switches are in four groups: Head tags, Feeds, Feeds: turn off and redirect and Headers. As elsewhere on the General screen, changes save automatically a couple of seconds after you make them.

There is no "recommended" preset. Each switch has a line saying what it removes and when to keep it, repeated below with what to check first.

Head tags

In the page source, the first five look like this before they are removed:

<link rel="shortlink" href="https://example.com/?p=123" />
<link rel="https://api.w.org/" href="https://example.com/wp-json/" />
<link rel="EditURI" type="application/rsd+xml" title="RSD" href="https://example.com/xmlrpc.php?rsd" />
<link rel="alternate" title="oEmbed (JSON)" type="application/json+oembed" href="https://example.com/wp-json/oembed/1.0/embed?url=…" />
<meta name="generator" content="WordPress 6.x" />

Notes on three of them:

• Emoji. Current browsers show emoji without the script. The switch also removes the hint that tells browsers to look up the address emoji images are loaded from, and leaves emoji in feeds as text.
• Generator. Removing the version number is tidiness, not protection. It does not make an out-of-date site safer; updating does.
• Reply links. Each comment's Reply link is a separate address that crawlers fetch, one per comment. With the switch on, the link points to the comment form on the same page. Replies still thread when your theme loads the WordPress comment-reply script, which most themes do. In a browser without JavaScript, or on a theme without that script, a reply is posted as a new top-level comment instead. Test one reply on your site after switching it on.

Feeds

The three switches in this group remove feed links from the head only. The feeds themselves stay available, and your main feed is never changed.

The first two are independent: removing the site-wide comments feed link leaves each post's own comments feed link as it was, unless you switch that off too.

Feeds: turn off and redirect

These three go further. Each turns a feed off: its address redirects (301) to the page it belongs to, and its link is removed from the head. Feed readers subscribed to it stop getting updates. Your main feed is never redirected.

On a large site, post comments feeds are one extra address per post, which is why this group exists. Three details:

• Only feeds in WordPress's own formats are redirected. A feed another plugin adds under a name of its own is left alone.
• Turn off post comments feeds applies to content that has comments. A feed of something else at a single item's address, such as the replies to a forum topic, keeps working.
• If your RSS feed text or anything else depends on your main feed, nothing here touches it.

Headers

Two limits, both stated on the switch:

• A pingback tag written directly into a theme's header file stays. The switch covers themes that print the tag through WordPress's standard check. If the tag is still in your page source afterwards, it is hard-coded in the theme's header.php, and removing it means editing the theme, preferably in a child theme.
• It does not turn off XML-RPC or pingbacks. Other sites that already know your pingback address can still use it. To stop accepting pingbacks on new posts, untick Allow link notifications from other blogs (pingbacks and trackbacks) on new posts under Settings → Discussion.

Check the result

1. Before changing anything, open a post in a private window and view the page source. Note which of the tags above are there.
2. Switch on what you have decided to remove.
3. Clear your page cache if you use one, reload the post and view the source again.
4. For headers, look at the response headers in your browser's developer tools (Network tab) or run curl -I https://example.com/a-post/. The Link and X-Pingback lines should be gone if you removed them.
5. For a redirected feed, open its address. You should land on the page it belongs to.

Should you use it at all?

Removing these tags does not improve rankings, and we make no claim about speed: each tag is a line or two of HTML. The case for cleanup is narrower:

• Fewer addresses for crawlers to fetch. Reply links and per-post comment feeds are real URLs, one per comment and one per post. On a site with thousands of posts and active comments they add up to a large number of addresses with nothing new in them.
• A head you can read. When you inspect a page to debug a meta tag, less noise helps.
• Not announcing what you do not use. If nothing publishes to your site remotely, the discovery links for remote publishing serve nobody.

If you are unsure about a switch, leave it off. A small site loses nothing by leaving all fourteen off. A tag-by-tag walk through the WordPress head is in cleaning up the WordPress head.

Developer filters

hydrogen_seo_crawl_cleanup

Filters the switches in effect for the current request, after the saved settings are read. One argument: $switches (array of switch name to boolean). The names are shortlinks, rest_links, rsd_links, oembed_links, generator, emoji, replytocom, comments_feed_link, post_comments_feed_link, extra_feed_links, redirect_comments_feed, redirect_post_comments_feed, redirect_search_feed and pingback. The switches are read once per request, late on init, so add the filter in a plugin or in your theme's functions.php, not inside a later hook.

// Always remove the generator tag and emoji script, whatever is saved.
add_filter( 'hydrogen_seo_crawl_cleanup', function ( $switches ) {
    $switches['generator'] = true;
    $switches['emoji']     = true;
    return $switches;
} );

hydrogen_seo_crawl_cleanup_feed_redirect

Filters where a switched-off feed redirects to. It runs only for a feed whose switch is on. Arguments: $target (string, the address) and $type (string: comments, post_comments or search). Return an empty string to serve the feed instead of redirecting, or another address on the same site to redirect there. An address on another site is not followed.

// Send the site-wide comments feed to the blog page instead of the homepage.
add_filter( 'hydrogen_seo_crawl_cleanup_feed_redirect', function ( $target, $type ) {
    return 'comments' === $type ? home_url( '/blog/' ) : $target;
}, 10, 2 );

Performance

With every switch off, nothing is hooked and your site's output is unchanged. With switches on, the plugin unhooks WordPress's own callbacks or answers a yes-or-no filter. No database query is added to any page.

FAQ

Will this break the block editor or plugins that use the REST API?

No. Remove REST API links removes the link that announces the API, not the API. The editor and plugins call it by its address and keep working.

Does removing the generator tag hide my WordPress version?

From the page source and feeds, yes. The version can still be worked out in other ways, so treat this as tidiness and keep WordPress updated.

I switched on the pingback switch and the tag is still in my source. Why?

Your theme has the tag written straight into its header file. The switch cannot remove that; see Headers above. The X-Pingback header is removed either way.

Will turning off a feed lose subscribers?

Readers subscribed to that particular feed stop getting updates, since its address now redirects to a page. Your main feed is never touched, so people subscribed to your posts are not affected.

Do threaded comments still work with the reply-link switch on?

Yes, on themes that load the WordPress comment-reply script, which most do. Without it, or with JavaScript off, a reply is posted as a new comment. Test one reply after switching it on.

Does Crawl cleanup change robots.txt or noindex anything?

No. It removes tags, headers and feed addresses. Indexing is controlled under robots meta settings and the robots.txt editor.

How do I undo a switch?

Switch it off. Nothing is stored beyond the switch, and the tag, header or feed returns on the next page load once any page cache is cleared.

Related reading

• General settings
• RSS feed text
• Author and date archives
• Filters reference
• Crawl budget in the glossary
• Cleaning up the WordPress head: what each tag is and whether you need it