The 404 Monitor
Log the URLs visitors and bots hit that return a 404, without storing IP addresses, and redirect them from the list.
On this page
- Why log 404s on the site itself
- Switch it on
- What is recorded
- Query strings and the home page
- Storage and clean-up
- Work through the list
- Deciding what to do with each URL
- Privacy
- Performance
- FAQ
- Why is the 404 Monitor off by default?
- Does the 404 Monitor redirect anything automatically?
- Why is a URL I fixed still in the list?
- Will the log show 404s for images and files?
- Related reading
The 404 Monitor (Hydrogen SEO, under Tools) records the URLs on your site that real requests hit and found missing. Each broken URL is one row with its hit count, when it was first and last requested, the last referring page and the last user agent, and you can turn a row into a redirect without leaving the screen. It is off until you switch it on, it stores no IP addresses, and old entries are deleted automatically. The screen arrived in version 0.0.27 and is free.
Why log 404s on the site itself
Search Console's Not found (404) report lists the broken URLs Googlebot found. It cannot show you a visitor following a mistyped link from a forum, a newsletter link to a post you renamed last year, or an image a partner site still hotlinks. Those requests reach your server, though, and the 404 Monitor writes them down. That turns "we probably have some broken links" into a list sorted by how often each one is hit.
Switch it on
- In wp-admin, open Hydrogen SEO → 404 Monitor (under Tools).
- Turn on the switch in the 404 Monitor card. It is saved straight away.
- Under Keep entries for, choose 7, 30 or 90 days. The default is 30, and this is saved as you choose it too.
From then on, every front-end page request (GET or HEAD) that ends in a 404 is recorded, after redirects have had their chance to run, so a URL your redirects already handle never shows up. Give it a few days of normal traffic before drawing conclusions.
What is recorded
Requests for the same URL are grouped into one row:
| Column | What it holds |
|---|---|
| URL | The requested path, without its query string or fragment. Accented and non-Latin characters are shown decoded, as /café-menu/, and /café-menu/ and /caf%C3%A9-menu/ count as one entry |
| Hits | How many times it has been requested since it was first logged |
| First seen | When the first request was logged |
| Last seen | When the most recent request was logged |
| Last referrer | The page the most recent visitor came from, when the browser sent one, also without its query string |
| Last user agent | The browser or bot that made the most recent request |
No IP addresses are stored, and nothing identifies a visitor. The referrer and user agent are kept only for the latest hit on each URL, which is enough to see where a broken link lives and whether people or bots are following it.
Requests to wp-admin, the REST API, AJAX, WP-Cron and XML-RPC are ignored, as are POST requests such as form submissions. They are not pages visitors read, and a missing endpoint there is a developer's problem rather than a broken link.
Query strings and the home page
Requests are grouped by path, with the query string removed, so /old-menu/?utm_source=newsletter and /old-menu/?ref=fb are one row: /old-menu/. A scanner that varies the query string on every request cannot add a row per request either.
The home page is the exception. With Plain permalinks, /?p=123 is the only thing that identifies a page, so on the home path the query string is kept and each variant gets its own row. Those rows have no Redirect button: the screen explains that a redirect there would redirect the home page itself. Fix them at their source instead, or switch to another permalink structure.
Storage and clean-up
Entries live in a database table of their own, created when you first switch the monitor on, so the log never bloats your options or post tables. Every hour, entries that have not been hit within the retention period are deleted. The log holds at most 5,000 addresses. The cap is checked while logging, about once every 100 new addresses, as well as by the hourly clean-up, so a bot hammering thousands of junk URLs cannot fill your database; the least recently seen addresses make way. Deleting the plugin removes the log, on every site of a multisite network.
For developers, hydrogen_seo_404_monitor_max_rows changes the 5,000 cap, and returning false from hydrogen_seo_404_monitor_record (which receives the requested URL) skips logging a request.
Work through the list
The Missing addresses card lists the log, with the number of addresses at the top.
- Sort by Most hits to see the broken URLs that cost the most visits, or by Most recent to see what is breaking right now.
- Search addresses for a path, such as
/wp-content/uploads/to find missing images, or a slug you recently changed. - Redirect a row when the content lives somewhere else: enter where the address should go, keep 301 permanent or choose 302 temporary, and click Create redirect. The form shows the path followed by (any query string): the rule matches the path without its query string, so every variant of it is redirected, and it is added to the Redirection Manager like any other rule. A row whose address now redirects shows a Redirected 301 or Redirected 302 badge with the target instead of the button, so you do not redirect the same URL twice.
- Delete selected removes the rows you have ticked once you have dealt with them, and Clear log empties the whole log (after a confirmation) when you want to start counting again after a clean-up.
Deciding what to do with each URL
Not every row deserves a redirect. A quick way through:
- A real page moved or was merged. Redirect it to its successor with a 301.
- A typo in someone else's link, such as
/pricng/. Redirect it to the page the link clearly meant. - Your own page links to it. The last referrer is on your domain. Fix the link in that page, and add a redirect too if the broken URL also has outside links.
- The content is gone and nothing replaces it. Leave it as a 404. Redirecting it to the home page tends to be treated by Google as a soft 404 anyway; the reasoning is in 301 vs 404.
- Junk requests from bots, such as probes for
/wp-login.php.bak,/.envor admin panels of software you do not run. Ignore or delete them. A 404 is the correct answer, and redirecting them helps nobody.
The full method, including Search Console and site audits, is in find and fix 404 errors.
Privacy
Because no IP address or other identifier is stored, the log records which URLs failed, not who requested them. Query strings and fragments are dropped from both the address and the referrer before anything is saved, since that is where tracking IDs, email addresses and session tokens tend to travel. The latest referrer and user agent per URL are deleted with the entry when the retention period ends. Mention the log in your privacy policy if your policy lists what your site records.
Performance
Logging a 404 is one small database write on a request that was going to show an error page anyway. Pages that load normally cost a single check of whether the request is a 404, and while the monitor is off nothing is hooked at all. If your host serves cached 404 pages from a CDN or page cache without reaching WordPress, those hits never reach the monitor, so counts on heavily cached sites are a lower bound.
FAQ
Why is the 404 Monitor off by default?
It writes to your database on every 404, and not every site wants that. Switch it on when you want the data; switch it off again whenever you like. Turning it off stops logging; entries already recorded are still removed when their retention period ends, or straight away if you clear the log.
Does the 404 Monitor redirect anything automatically?
No. It records and shows. You decide which URLs get a redirect and create each one from its row, choosing 301 or 302. For automatic redirects when you change a slug, use Auto Redirection in the Redirection Manager.
Why is a URL I fixed still in the list?
The row stays until it is deleted or ages out. After you redirect it, the URL no longer returns a 404, so its hit count stops growing and its last seen date stops moving. Delete the row if you want the list tidy.
Will the log show 404s for images and files?
Yes, when the request reaches WordPress. Missing files that your web server answers directly, without loading WordPress, are not seen.