Your 404 Log Is a To-Do List: How to Read It and What to Fix First

A log of the 404s your site serves is useful only if you can tell, in a few seconds per row, whether a URL needs a redirect, a link fix, or nothing at all. Three columns do most of the work: the hit count says how much a broken URL costs, the referrer says where the broken link lives, and the user agent says whether a person or a program is asking. Read them together and a list of several hundred rows usually comes down to a dozen that matter.

This is a method for that reading. It assumes a log that groups requests by URL and keeps a count, first and last seen dates, and the latest referrer and user agent for each, which is what Hydrogen SEO's 404 Monitor records. Server access logs carry the same information per request, so the method works on them too with a little more sorting.

Start with the hit count, but do not stop there

Sorting by hits puts the most expensive broken URLs at the top. A URL requested forty times a week is forty people, or forty crawler visits, hitting a dead end.

The count needs the dates beside it to mean anything. Forty hits since the first seen date three months ago is a slow leak; forty since yesterday is something that just broke. Two patterns are worth recognising:

• First seen is recent and hits are climbing. Something changed on your side: a slug rename without a redirect, a deleted page that was still linked, a menu pointing at a draft, a migration that missed a path. Fix these first, because they are new and they are growing.
• First seen is old, hits arrive in a trickle. An old link somewhere on the web, a bookmark, an email from years ago. Worth a redirect when a replacement exists, rarely urgent.

A high count with a last seen date weeks ago is history. Whatever sent that traffic has stopped.

The referrer tells you where to fix it

The referrer is the page that linked to the broken URL, when the browser sends one. It splits the log into three piles.

Your own domain. You built this dead end. The right fix is editing the linking page so the link points at the correct URL. A redirect would also work, but it leaves a broken link in your content forever, adds a hop for every visitor, and hides the problem from the next person who audits the site. If the same broken URL also gets outside traffic, fix the link and add the redirect.

Another site. You cannot edit their page, so the fix is on your side: a 301 from the broken URL to whatever it was meant to be. If the link is a typo, such as /prcing/ for /pricing/, the intended page is obvious. If it points at content you removed, redirect to a genuine successor when there is one and leave the 404 when there is not. For a link from a site that matters to you, it can also be worth asking them to correct it.

Empty. Many requests arrive with no referrer: typed addresses, bookmarks, links in apps and emails, privacy settings and browsers that strip it, and most bots. An empty referrer with a browser user agent and steady hits often means an old email or a printed address. An empty referrer with a bot user agent usually means nothing at all.

The user agent tells you who is asking

The user agent string names the software making the request. It can be faked, so treat it as a strong hint rather than proof, but it sorts the log quickly.

Browsers (strings mentioning Chrome, Safari, Firefox, Edge): people. Their 404s are the ones that cost you readers and customers.

Search engine crawlers (Googlebot, Bingbot, YandexBot and others): a crawler requesting a dead URL found it somewhere, in an old sitemap, an external link or its own memory of the page. One visit is normal. Repeated visits mean something keeps pointing it there; check your sitemap and internal links before anything else.

AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and similar): these fetch pages for AI models and AI search. Treat their 404s like a search crawler's. If one keeps requesting a URL you moved, a redirect is the way to point it at the current page.

Scanners and scripts (empty or generic strings, command-line tools, odd names): requests for /wp-login.php.bak, /.env, /phpmyadmin/, /xmlrpc.php variants or admin panels for software you do not run. These are automated probes looking for weaknesses. A 404 is exactly the right answer. Do not redirect them, do not let them dominate your list, and delete them in bulk when they crowd out the real problems.

Patterns that point at a single cause

Some rows are symptoms of one underlying problem, and fixing the cause clears them all at once.

• Many 404s under /wp-content/uploads/. Images or files were deleted, or a migration did not copy the uploads folder. Look at the referrers: if they are your own pages, restore the files or edit the posts; if they are other sites hotlinking your images, a 404 may be fine.
• The same slug under several paths, such as /blog/spring-menu/ and /2024/04/spring-menu/. The permalink structure changed at some point. If many old addresses are affected, a single rewrite rule at the server for the old pattern covers them all; otherwise redirect the ones that still collect hits and let the rest age out.
• A trailing segment that should not be there, such as /spring-menu/amp/ or /spring-menu/feed/. A feature that used to exist, like an AMP plugin, has been removed and old links still point at its URLs. Redirecting the busiest of them to their base URL is reasonable.
• Query strings on a valid-looking path (in server logs, which keep them). WordPress usually ignores unknown query strings on a page that exists, so a 404 here means the path itself is wrong. Read the path, not the parameters.

A worked example

Five rows from a hypothetical log, sorted by hits, and what to do with each:

The first row is the expensive one, and it is self-inflicted: a renamed page still linked from the site. The third is noise. The fourth is a judgement call that depends on whether the image still exists. None of them needed more than a glance at three columns.

Keep the routine short

A weekly look is plenty for most sites:

1. Sort by last seen and read the top of the list. Anything new since last week gets fixed now.
2. Sort by hits and handle anything with a real referrer or a browser user agent that you have not dealt with.
3. Delete the scanner rows and the rows you have fixed, so next week's list starts clean.

After a migration, a redesign or a bulk content clean-up, check daily for a week. That is when broken URLs appear fastest.

Privacy is not in the way

None of this needs visitors' IP addresses. Which URL failed, how often, from which page and with which kind of software is enough to fix every problem described here. A log that leaves the IP out records broken links, not people, which keeps it out of the way of most privacy obligations and makes the decision to switch it on an easy one.

Doing it with Hydrogen SEO

From version 0.0.27, Hydrogen SEO's 404 Monitor (under Tools) keeps exactly this kind of log. It is off by default. Once on, it groups 404s by URL with the hit count, first and last seen, last referrer and last user agent; stores no IP addresses; drops query strings, except on the home page; ignores wp-admin, REST, AJAX, cron and POST requests; caps the log at 5,000 addresses; and deletes entries older than 7, 30 or 90 days. You can sort by hits or last seen, search, delete rows, and create a 301 or 302 from a row in one step, with already-redirected URLs marked. Settings and details are in the 404 Monitor guide. For the separate question of whether a dead URL should be redirected at all, see 301 vs 404.