There are some very different schools of thought out there regarding 404 error code pages. Some SEOs recommend:
- Never allowing them - 301'ing every error page back to the home page or an internal category level page to preserve the maximum amount of link juice (in case someone links to a broken URL)
- Letting any erroneous/mistyped URL 404.
- Something in between - 301 some kinds of 404 pages and not others.
I'm generally in this last group. I think there are times when it pays dividends to let a URL 404, both for accessibility and search engine reasons. I also don't think it's intuitive or semantically accurate to 301 every 404 page on the site - it certainly pays to build great custom 404s (good piece with examples on that here), but to simply have your homepage appear when a URL is mistyped or a link breaks doesn't send the right message to users or search engines.
When faced with 404s, my thinking is that unless the page:
A) Receives important links to it from external sources (Google Webmaster Tools is great for this)
B) Is receiving a substantive quantity of visitor traffic
and/or C) Has an obvious URL that visitors/links intended to reach
It's OK to let it 404.
Recently, though, Lindsay and I were faced with a tough call on a consulting project. The client has a site that receives a ton of search queries, many of which map to their category and subcategory level pages (which are more landing pages than search query pages, but also serve to address the search keywords). The client also has a number of search pages that have no content (either because they're for mis-typed, nonsense or mis-spelled searches or because they simply don't have content for those terms). Some of these pages earn links, some get a moderate amount of traffic and up until recently, they've essentially existed as error pages that resolve with a 200 code.
What to do?
Our conundrum contained a few critical elements. We don't want the search engines wasting bandwidth crawling and indexing junk pages (especially since the site is monstrous and needs that crawl/index power to flow to the right sections). We also don't want users to have a bad experience and while the error pages effectively communicate the right message (there's no results for this query), semantically the pages should really 404. Finally, of course, we don't want to waste any of that precious link juice that's flowing to some of them.
The solution turned out to be a compromise - we'd 404 the pages, but keep track of those that earned links and any substantive level of traffic and try to build better experiences for those pages (sometimes a 301 to a sub-category page, sometimes to a results listing and sometimes we'll actually add content to those pages and make them resolve). We hope that this lets us have our cake and eat it, too.
We'd love to hear your thoughts around 404s and SEO in general, as well as on this specific scenario (and others like it). 1000s of SEOs are smarter than 2 :-)
I think 301's can be more confusing to searchers then 404's. It's like asking a taxi to take you somewhere (that the driver knows doesn't exist) being taken round the block and dropped off where you started, much better to just be told that the location doesn't exist.
To use the same analogy though, if you ask to be taken to TSB (a UK bank that was emalgamated with another) and they take you to Lloyds TSB because that's where they know you really meant, then that's a great customer experience.
I like the metaphors you used in that, may actually use the 301 to home page one as most web devs don't see that as a problem...
Or like asking the store clerk for a pack of smokes and being punched in the face.
..."Camel Menthols, please." - !@*KAPOW*@!
..."Ouch, I just got 301'd in the face!"
On a more serious note. 301's provide a tool for funneling relevant traffic to the appropriate pages on a website. Just because it's available doesn't mean you should use it for every situation.
Valid uses could be (not limited to):
- url has changed
- people typically misspell the url
Custom 404's actually help me. I'm not sure about you guys, but if i land on a custom 404 page - I usually stay around to try to find what I'm looking for. If I'm redirected to the home page, I usually get frustrated.
It's like - Game Over, try again. Not good on my websurfing ego.
oh also,
I listened to Matt Cutts site review twice yesterday and the recurring theme was user experience. If the user has a positive experience your site will become popular and your rankings will increase. It's easy as SEO's to get caught up in the how we can manipulate things instead of focusing on the user and really making sure we are first doing what is in their best intent and 2nd maximizing our results. I liken it to the super store example; sure the store could increase sales of nudy magazines by placing them in the front of the store but is that really the experience you want to deliver?
I'm surprised no one has mentioned 410 error codes yet.
When a page is completely gone, and it does not make sense to 301 it to a different page of the site, then a 410 may be the most appropriate error, instead of a 404. Here's the difference:
404 = Page Not Found (but may reappear later)
410 = Permanently Gone
BUT! Matt Cutts has said that Google treats 404's and 410's the same, so from an SEO perspective, I don't think it makes a difference. (not sure about the other engines though)
If you want to 410 a page, you can do it on *nix based hosting by adding this to your .htaccess file:
redirect 410 /folder/file_that_is_gone.html
In case of 410, the search engine will drop immediatelly the page from the index, while at 404 status code it will drop after 30 days only (normally).
In case of URLs with very short lifetime (some days) i'll suggest to use 410 status code, always.
But i'm really interested on "official suggestions" by Googlers about these cases, illustrating with some examples, like classified ads websites, real estate websites or price comparison websites for example.
I agree with Rand, there is no usual way and the best solution is to balancingbetween these 3-4 available solutions to solve the problem of "outgoing urls" :)
There are scenarios where the status code is very important for both SEO and UX:
A classifieds website where users can post ads, once validated and ad is displayed on the website and can be hidden or deleted by its owner, the site owner also allows users to define url shortcuts so that example.com/ad?id=123 becomes example.com/ads/seo_expert_in_morocco.
200 : If the ad is available to the world
301 : when redirecting the visitor from a normal url to the shortcut url
302 : ad is hidden by its owner, but he may choose to show it at any time, so we redirect visitors to the user classifieds listing page and show them a nice error message
404:
410: This one is for junk, ads flagged by the community or illegal things that the owner didn't notice at first but deleted later.
Question - I know all (or at least a lot) of the statements about '404s don't hurt your rankings' but I wonder if that is true. I think there might be several ways it hurts in natural search:
1. Too many = quality issue. Let's say the average number of 404s is about x % of your visits, so significantly more visits are to 404s, this is a quality issue = low quality = lower ranking (?)
2. Bounce rate. Usually - especially with not customized 404s - a lot more people will bounce off these pages than other pages, and google seems to take bounce rate as a quality signal as well.
3. Lower / no rank for keyword. Would google not take the (old) content completely out of the index? Let's say they hit the same page 3 times, would they not take the page and all keyword rankings for this page out of the index to keep their results relevant?
So, while they might not 'punish' the overall site, there are still consequences I would expect.
I would definitely avoid 301'ing any and every non-existent URL... don't want to send a low quality signal to Google and other engines if they are testing how 404's are being handled by requesting a noexist page.
Of course the smartest thing here is to dig in to understand the site, rather than making a blanket determination on how to handle.
- smart to understand inbound (or even internal) linkage to harvest existing high value pages.
- I would also avoid serving up "empty search" pages on-site b/c these are also perceived as low value pages and another variation of search-in-search that Google especially is and will continue to target more and more to eliminate.
- for misspellings and alternate spellings, I'd tried to find a way to integrate that into the relevant page to try to help it supplant the other pages... especially in place of some links to those pages, which searchers may be just as inclined to hit the back button as click forward into the site.
- once important pages, based on phrases and inbounds links, have been harvested in some way, I'd simply let all the low value pages simply 404 to get them out of the index and rather than spending more resources trying to use them, which will probably not deliver enough value for the resources spent, or hinder the site overall by sending a lower quality signal.
- on those pages that do 404, I'd still try to serve up at least a simplified sitemap/search box, "did you mean?" and/or related links to help human visitors.
One method we have used in past is to let the user decide (sort of)...
Imagine a 404 page with a giant search box smack dab in the middle. The user searches from there and then chooses the page they want to go to.
We record that path and, over time, can find patterns where people are regularly navigating to the same alternative page.
Human intervention is only needed when the system is ready to recommend to the webmaster that a currently non existent 404 page should be 301 redirected because of that consistent pattern of navigation.
In my opinion, when you are making a decision about your website you should make one that makes sense for a user and ignore the search engine. Google strives to provide accurate information from the point of view of a user.
For example:
If you are a user and you want to go to domain.com/happy_hour and you type domain.com/hapy_hour you should get a 404 page. This 404 page should have a sitemap or search or even a "did you mean". This 404 page tells the user that they did something wrong with the url. Redirecting to the home page would more than likely confuse the user.
Thats my two cents, if a site is useless to a user it doesn't matter how many visitors a search engine brings in.
I agree with you. Redirecting the user to the home page will not alert the user of any notification that they have went to a wrong link and will most likely confuse them. This can lead to a confused and unhappy visitor and give a bad reputation for your website.
On this subject, I prefer to show the 404 page if done in the right way. I found a TED talk that explains how you can change the bad experience of a 404 page:
https://www.ted.com/talks/renny_gleeson_404_the_story_of_a_page_not_found.html
We always use custom 404 pages, regardless of the site, to direct them to the homepage or sitemap or site search.
On sites where we know there are certain pages that should be 301'ed instead, we'll do that.
I think you hit the nail on the head with "try to build better experiences for those pages"... that's essentially what it's all about. If it makes sense to 301 a page, then do that. If it makes sense to build a page out, then do that. 404 errors aren't just messages that the user screwed up, they're messages to us that we've screwed up somewhere as well, and we have the opportunity to fix it.
Rand,
I am of the same opinion that a hybrid solution will usually lead to the best results for people and spiders. But what about using a mod-rewrite and canonical tags? You said some of the queries had no results. If you consider that some of those queries might just have valid results in the future, or if you recommend to the client to develop the results, then a mod-rewrite with a canonical tag can be implemented now and removed later. Of course it depends on the specifics of the client site.
Hi, I have a different case on 404. I use a custom 404 for all my out of stock product pages but on the same product url that means, the url of the page remains same when a product goes out of stock and this url shows a custom 404 error message.
Now in webmaster all these urls which went out of stock, its showing 404 not found and this number is 400K and increasing as daily around 1000 products go out of stock.
What is the best practice to handle this situation. Kindly suggest.
I prefer to disallow empty search pages and redirect static pages, though I always redirect to the subdirectory that the old page belonged to, so if someone redirects from a missing blog article, they'll just end up at the main blog page, which doesn't seem to cause any problems from a UX perspective.
Custom 404 page is best solution for me. But, I'm handling broken links with different method. I'm focusing on each and every broken links with help of Google webmaster tools. If I found any old page which have certain page rank so I will use htaccess to redirect it on specific root category or home page. It may help me to improve user experience and save my page rank.
Hi Lindsay,
Thought that this was a great post, and one that I have sent around in an email to numerous of people.
What would you suggest for parameters that automatically populates a drop down list with items.
www.domain.com/example.html
www.domain.com/example.html?jptype=popular
These are being indexed and causing major duplication throughout the website. I was under the impression that we could add a rel="canonical" into the ?jptype=popular URL but noticed that they are not shown within the CMS.
Any suggestions?
Daniel
I need help here Randfish if you can please. I made a mistake on blogspot by puting "/" after html when i was trying to make anchored backlinks . Problem is i did it when i was doing article distribution for several hundreds article directories and blogs. Is it possible to do 301 for those kind of pages. I never did 301 so i dont know how would i do this when i cant make a page on blogger that would end with "html/" . Tnx
Hello,
Could you please answer me what is the exact name of "404 Not Found Page"
i have one question come up in my mind related 404 status
for ex.: www.abc.com index in Google with 250+ pages now www.abc.com change business so delete all old content from website and keep same domain for future product but www.abc.com is already index with 250+ pages in Google, so while any visitor or bot come it will redirect on 404 page and that site have 250 pages with 404 error code, and new updated website have only 30 pages with quality and relevant content but that website have so many 404 pages so
1. Too many = quality issue. Let's say the average number of 404s is about x % of your visits, so significantly more visits are to 404s, this is a quality issue = low quality = lower ranking (?)
2. Bounce rate. Usually - especially with not customized 404s - a lot more people will bounce off these pages than other pages, and google seems to take bounce rate as a quality signal as well.
3. Lower / no rank for keyword. Would google not take the (old) content completely out of the index? Let's say they hit the same page 3 times, would they not take the page and all keyword rankings for this page out of the index to keep their results relevant?
While they might not 'punish' the overall site, there are still consequences I would expect. How to solve this kind of issue, does Google really remove all 404 pages from index or not.?
Google 404 Pages are the best.
There are 2 webpages, which both have 3000 links to my page, all of them link to 2 different non existent folders. In GWT i have about 6 000 not found links. What would be the practice?
To permanent redirect these folders?
Leave them as 404?
I have disavowed that stupid domain on Jan 3, 2013 - no result so far.
I'm looking for best practice about website A, whit categories/sub pages X, Y, Z. But, category Z is removed permanently, because of inapropriet content(sexual) google don't like it. The category Z has many links to it, but there is no such category/pages.
Questions is what to do:
1) Return 404 header, and show custom 404 page with links to home page?
2) Redirect with 301 to home page to gain juice/popularity?
Hi,
I have a very relevant and urgent question that i'm slightly confused about.
If a site doesn’t generate 404 errors does this affect good SEO ??
Eg… on www.alpro.com/uk/drink if you type in anything after this within the url is doesn’t give a 404 error ??
So www.alpro.com/uk/drink/WHATEVER (does not generate a 404 error)
according to me this isn't good SEO practice but would really appreciate some opinions/advice?
Thanks
Ankita
What to do with all the 404 we get with real estate.. when 100 homes sell each day.. and another 100 come on the market.. is it ok to make sure the page says this home is sold.. and send a 200 to the seach engine so they do not remove the page.. we by law must remove the photo and information about the home .. so is it ok to just add similar homes and mod the 404 to a 200 and keep building 100 pages a day forever.?
Well, Plz help me, i have lots of 404 errors in my website
https://www.toparticledatabase.com/ . I also use 301 redirect but 7 errors still remains out of 93 errors. What to do....
Wasn't there a Whiteboard Friday about this same topic not too long ago? And are these responsible for a lot of traffic or are we getting at the minutia of SEO at this point? What's the marginal gain, I wonder.
Disagree. The search results PAGE is found whether there's 0 or 1M results. What do Google or Yahoo return as a status code when they show no results?
When I was new to SEO I had it in my head "404 and Search Engines bad" and set a site to return 200 ok on every page (this was a long long time ago). I'd really not recommend for so many reasons (user experience, accessibility etc) but it never actually caused any problems.
If I was doing this work again I'd create a great 404 page with search, common links etc and use webmaster tools to see which pages were 404ing. Those pages that we're 404ing and appeared valuable I'd then 301 redirect.
Obviously this would only work for small and medium size sites but I believe it’ll give the most search engine value and best user experience.
Hi Rand,
Personally I too believe in something in between 301 & 404 pages.
Depending on situation if I feel the page is having some good link juice and don't want to waste it, I use to 301 that URL to most relevant existing page on my site (Not on Homepage or Sitemap), if I found no relevant content page exist on my site then place re-direct to (Home or Sitemap page).
While 404 - Error page is also very much needed over site to show for bad links. It's like we need to have mix of 301 & 404 page based on our objectives.
Thanks :-)
I also have a few links that are to pages that don't exist. I've been redirecting them. I'll reconsider your solution but may not be worth the trouble for just a few pages.
Rand, personally I would of tried to find a way to automate some results on those pages if they were doing so well (while doing nothing! :)). For for mispellings, link it to the actual page, less popular terms, link them to the more popular versions, etc.
Would be a lot of work but would work so well meaning the SE's wouldn't get pissy and would still be a fairly good customer experience as your still pointing them in the right direction :)
Exactly.
In this situation I would suggest trying to determine related tags or other taxonomy, and pull in related or high priority content, plus links to the related tags.
From the description it sounds like something akin to Mahalo who have recently started improving their "no term" pages, but they are still messing it up ;)
Hi Rand,
I believe to use 301 redirect for all 404 pages and send visitors to sitemap. This is the page where they will find the resource of their choice. I dont think it will hurt from search engine point of view.
What do you think?
My two cents:
If it makes sense to redirect the user to the resource they really were looking for do so. If not serve the 404.
The first part of that requires you to know the searchers intent. Our job is to provide a great user experience much like manager of a store. If the user asks for something we don't have we can reply, we don't have that item. OR if we have a comparable item that meets the user's needs and matches their intent direct them to that resource.
I have an interesting situation.
My previous blog on my same domain got hacked a couple of years and hadthousands of mega spammy links injected into not only comments but even replaced some old blog posts! (so ALWAYS update your WordPress themes everyone.)
While this was happening, seemingly overnight, my PageRank soared as several spammy sites linked to mine. Eventually my little site gained an average PR of 6! Then almost immediately became removed from the Google index.
Sinse then I have completely redisigned the site (survivestyle.com) and have essentially removed the old blog.
Now Google Webmaster Tools says that these are the only 404 pages on my site. At this point, I don't think I want to redirect them as these made my site looks spammy.
What does everyone else think? These are NOT links I would want to 301 right?
I would think 404 pages are best, maybe custom 404 pages that are a bit more useful with a search box, popular dynamic links etc...
Once you see the 404 pages building up I would do block 301 redirects to a category product that is as related as possible so if you get manually audited by their search quality team they can see you are not trying to pump up random pages but are in fact improving the user experience...
dont quote me on that the spam team maynot crack the shits if you have thousands of these for every possible product/term but it can be useful when dealing with limited control over inbound link requests...
I too have always enjoyed your writing and will continue to be a fan wherever you decide to go next!
Best of luck Rebecca. Although with your skills, I'm sure you don't need any luck. :)
Hey weird. I was reading up on 404 pages and saw my comment here. It was definitely meant for another post where Rebecca Kelley was saying her goodbyes to Mozzers. I have no idea what happened. FYI.
can you guys give me a hand and explain what the guy Phil Payne is saying about the Freedom website.
https://www.google.com/support/forum/p/Webmasters/thread?fid=4e269698d49aa14b0004749001e99ed9&hl=en
My site has over 7 million pages and its only getting 6k indexed now. I need some help understanding why and he said something about the 404's.
One of my 404's is on that list. =) When I worked for that agency we needed some link bait, and that got us a good bit of links (which we then linked from that 404 lander to our top 5 important pages).
It is the Birds on the Wire one.
https://www.smashingmagazine.com/2009/01/29/404-error-pages-one-more-time/
Glad to see my concept payed off. Don't work there anymore anyways =).
If it is simply a new URL with the same content, I 301. If the URL and content is simply nonexistent, I keep the 404 for the most part. The only exception would be if the page was a very substantial and prominent page, or if there are tons of links associated with it.
In my experience, though, a custom 404 is great. With the site I currently do SEO for, our error pages have exceptionally low bounce rate because people are willing to stick around to find the info. It depends, however, on the 404 page itself (design, user-friendlyness, etc) and your audience and how savy they are with the internet.
That's my two cents...
I think no magic solution for this issue.
as a web engineer I think we have two aspects:
-the (normal) user experience (persons that are using the brwosers interactively)
- the SE robots experience (for this point I dont care so much about bad bots, I care only about SE).
So the best solution (for me of course), is to distinguish between http clients requests...Is it coming from (normal) browsers? then the user experience underestimate a classic 404 error... we should give the persons the ability to go back or to go home or to go ... but not automatically.
n case it is a bot client http request (it never varies if it is from bad bots or from SE bots), I think we should not give every page the same treatment mechanism but I prefer to do either a 301 redirect for a specific page (depending on the page that is not found) or do a 410 error (also depending on the page that is not found).
n both cases, we should analyse our logs on the server and use the google specific tools that give us the pages that backlink the our "not found page".
we should also remeber that corrupted links are something bad in SEO... especially if spammers or maybe competitors backlink to a "not found" page on our domain...this will be a disaster if SE behaves like this
I generally use a custom 404 page which links to pages within the site. I like the idea of sending it back to the home page but don't use it as I agree with you, it is not good for search engines.
How it actually works?
Is page with status code 404 able to share it's juice with other links it points to? I'd say no, but am not sure.
I am going to agree with break787. I think once a 404 status is found by the search engines it clears all the link and page metrics for that page. So having links on that page to other pages on your site might not be as beneficial from a link juice standpoint.