URLs that return 404 pages have always been a source of headache for webmasters. This is especially true for websites that have tons of pages, since the chances of getting 404 errors increase this way.
The good thing though is that tools like the â€œCrawl errorsâ€ page on Google Webmasters Tools exist to make it easier to track such pages. All you need to do to find all the URLs that return a 404 error when crawled by Googlebot is to click on the â€œNot Foundâ€ tab of the â€œCrawl errorsâ€ page.
The question now is what to do with these pages? Should you 301 redirect them all or should you just let them be?
Obviously, the second option does not make sense. The fact that 404 pages are listed under crawl ERRORs signifies that things must be set to right. However, redirecting all the URLs that return 404 pages is also not the way to go, especially if you have thousands of URLs with such a problem. Instead what you should do is determine which of the URLs needs redirecting, and which you can just ignore.
Before going into detail as to which pages you definitely should redirect, I just want to make it clear that URLs that keep returning the 404 error when crawled by Googlebot will eventually be considered dead by Google and so not be crawled anymore. So while youâ€™ll still see the URLs you donâ€™t really care about still appearing on the â€œCrawl errorsâ€ page, you can just disregard them until they are no longer crawled by Googlebot.
As for the other URLS, the ones that you should pay attention to and redirect, how do you figure out what those URLs are?
I have already written about 404 Pages around two years ago, and my advice still remains the same. 404 Pages only when you get substantial traffic from that page, that page get inbound links from important sources, or is a typo of an important URL. As Joe Robinson puts it, â€œIf your 404 page is indexing or getting links from other sites, redirect it,â€ and if not, just let it 404.