Welcome to the third installment of Next Level! In the previous Next Level blog post, we shared a workflow showing you how to take on your competitors using Moz tools. We're continuing the educational series with several new videos all about resolving duplicate content. Read on and level up!
Dealing with duplicate content can feel a bit like doing battle with your site's evil doppelgänger—confusing and tricky to defeat! But identifying and resolving duplicates is a necessary part of helping search engines decide on relevant results. In this short video, learn about how duplicate content happens, why it's important to fix, and a bit about how you can uncover it.
[ Quick clarification: Search engines don't actively penalize duplicate content, per se; they just don't always understand it as well, which can lead to a drop in rankings. More info here.]
Hi Ellie
Follow is the default parameter, so there's no need to add "noindex, follow". Just use <meta name="robots" content="noindex">
I know it doesn't hurt to add follow, but I do recommended keeping SEO tech recommendations as simple as possible. I've worked with many front-end developer teams and have seen many unintended mistakes that ended up being serious issues for SEO.
People make mistakes, so keep this crucial parameter as simple as possible.
Google's recommendations are the same: https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag
Hey Gyorgy - I'm on board with keeping things simple! Thanks for your comment.
These tips are helping me a lot!
Thanks a lot for detailed explanation, I am regular user of moz tool and I enjoy using it. but somewhere I feel that when it comes to get list of competitor backlinks ahrefs is more powerful and provide more data compare to moz.
I'm glad you enjoy using Moz! Generally, we recommend using a wide variety of backlink tools to get the most illustrative picture of how your site's backlink profile looks. OSE and Ahrefs index differently and have different purposes. Ahrefs is good for quantity while OSE is great for finding higher quality links. Domain Authority is our own proprietary metric that is closely correlated to Google rankings, so our customers often use OSE to research influential sites to build links. :)
Thanks this is very helpful post ..i must say its very well written & explained .
https://www.travelwithsmile.com/TourPackages.aspx
Thank you very much for the article, it is serving me well. But I have a problem, I have an online store with children products and have read that google does not consider duplicate content. It may be this or I have to establish a canonical relationship?
Search engines don't actively penalize duplicate content, but it can affect your rankings because it confuses search engines when they're deciding on relevant content. Hopefully the 3 common solutions to duplicate content above can help you out! :)
after reading the three options I think the most suitable is to use a rel canonical , google this also advised. TY very much :)
Identifying duplicate content manually can be incredibly frustrating/slow so it's great that Moz have this system - also good to see so much advice for onsite optimisation (instead of just banging on about links!).
I saw a video with John Mueller who said that there is no actual 'penalty' for duplicate content (common misconception) - but it does create ambiguity and will reduce rankings as a result. I suppose when you get down to the basics the two concepts are effectively the same: your website/page will not rank as highly as it would if you eradicated your duplicate content issues.
Right, Simon! Thanks for clarifying. :)
Hi Ellie,
I am facing duplicate content issue specially for Ecommerce websites. Because one category has number of exact products with different measurements. For Example:
Green House Plastic Sheet is one Product and it has 20 different sub products with different measurement say 1.5 mt wide and 1 meter length, 1.5mt x 2mt length and so on. and all the sub products have similar content.
Show should I deal for this issue? Do I put canonical for all the sub products?
Could you please help.
Regards,
Sushil
The best idea is to use a canonical rel tag on the website referring to google that they are not duplicates and these are all referring to the parent pages.
Hi what e-commerce platform are you using? Ideally you should have a parent Product which has many Product options with each option being a measurement..
Hi, i hace an e-commerce with Wordpress + WooCommerce, and i think that the best way to avoid is input handly little diferences in the titles and content of products. But i have the same problem, and i don´t know if is sifficient with this practice, if anybody have better solution i´ll be happy to hear.
Hi Sushil
It has been discussed here on the MOZ blog before, find more info here
https://moz.com/learn/seo/canonicalization
Hope it helps!
Henry
Hi Ellie,
Thanks for the nice post. I just noticed most of the people still have confusion, which solution they should follow. My advice would be
Use "301 redirect" if you don't want duplicate page/domain to rank in SERPs, if you are using 301 redirect, its a clear signal to Google that this page/domain has permanently moved to new place, now you(Google) can ignore this(duplicate) page.
If you are using "Rel=canonical" it means that I want to keep both pages(original + duplicate) and don't treat this page as original source or preferred version.
For noindex,follow as said earlier in the post if you add "noindex, follow" if you have pagination problem, it works really well with paginated content or if you have a system set up to tag or categorize content.
Please correct if I am wrong.
Shahzad
Thanks for your indications, I wasn't aware of this
Great! and don't forget the rel="alternate" hreflang="x" for multilingual sites, critical for establish the different versions of a page.
Exactly what I was coming to add! One of our biggest clients has sites in 20(ish) countries and they would rank whatever site Google felt like for awhile. Not very good to have yoursite.com.sg ranking for Australia or yoursite.cn ranking for Qatar.
Hey Ellie hope your having a good weekend
We use moz tools and we are showing long url and duplicate content crawl issues, I would just like your guidance if we are fixing this in the correct manner
We have
ourwebsite.com.au/hotel_details?hotel=115073&cc=AUD&q='115073%2Foverview%3Flang%3Den%26amp%3Bcurrency%3DAUD%26amp%3BstandardCheckin%3D2%2F24%2F2015%26amp%3BstandardCheckout%3D2%2F25%2F2015%26amp%3BroomsCount%3D1%26amp%3Brooms%5B0%5D.adultsCount%3D2'&price=346.75&tday=1
So i have done the following with the hotel id variable
<link href="https://www.ourwebsite.com.au/hotel_details?hotel=' . $hotel_id . '" rel="canonical" />
which now shows
<link href=https://www.ourwebsite.com.au/hotel_details?hotel=1115073 rel="canonical" />
is this the correct approach to canonicalize it to the page for each hotel id we have with the correct title tags and so on so we index each hotel page accordingly or would I be better of bringing it up to the next level like so
<link href="https://www.ourwebsite.com.au/hotel_details rel="canonical" />
your feedback would be greatly appreciated
thanks
Alan
Does duplicate content actually cause a penalty? I was under the impression that it just affected the pages which were duplicates of each other
You're correct, it's not a penalty from the search engines. We'll be editing the post this morning to make sure that's clear. Sorry about the confusion!
Hello guys,
haha you made me smile when your "duplicate" appeared, nice one ;-)
Btw duplicated content is a really a problem that currently lot of websites are suffering, specially because Wodpress CMS loves to create new urls...
Currently i´m using Screaming Frog to detect duplicate content, but your suggestion sounds very good, i will give it definitely a try ;-)
Thanks for the info
Greetings
//Oliver
hello everyone,
Thnks Ellie for sharing such a nice post. I recently join moz it a very good platform for all about SEO. I am enjoying its amazing features. surely duplicate content is a big problem for websites i will give a try to above tool. There is a tool i use to defeat duplicate content is plagium.com.
Very clear post. Thanks a lot for share!
There is almost nothing more annoying than posting duplicate content. The problem with it is that it distracts from the new and important information that you have to post. It can also drag down search engine results on certain types of content because it makes a website look spammy. Avoiding duplicate content simply makes a website look more professional and like it is being handled by people who care about their readership. That is great for working towards building a loyal following. Redirect pages are one way to help control duplicate content. This sets up a system whereby if there is duplicate content on the site, it will all link to only the content that you actually want to have seen. It narrows down that excess content that is not relevant and may have already been shown. Likewise, there are codes which can be written into the website script which help rid it of duplicate content. These codes search for duplicates and eliminate them when they are found. These are important to place throughout the website code to make sure all duplicate material is eliminated.
Duplicate content is really a big SEO issue. Thanks for sharing your knowledge
I think we should follow the suggestions of Google and recommends that use rel canonical I think it is the best option
Worst of all is the duplicate content and even more when google grabs you by the hair and buries you.Soon Moz tools will be of vital necessity for every blogger in the world, now not be able to choose, just touch you use
Some content management systems are worse than others for generating duplicate content. Wordpress, I'm looking at you... So the starting point is always to choose your CMS carefully, hack your template to remove the worst effects, or if you can afford it hire someone who can code your a bespoke site with no duplicates.
I have one confusion. Is it necessary to use 301 redirect for subdomains. Like https://blog.xyz.com or https://www.blog.xyz.com which is better?
Hello Ellie,
I know Google really hates duplicated content so I have been trying my best to avoid the content from my blog from getting copied from elsewhere or someone. Their is actually a plugin that does disable right clicks and what not but I just wanted your opinion if this type of plugin affects Google robots from coming in or does it affect my ranking to have such plugin?
Would really appreciate your help, thanks so much Ellie!
Kind regards,
Randy Garcia.
Hey Ellie, could www.site.com/ and www.site.com be considered as a duplicate?
p.s. Dan, your Doppelganger is giving you the weird eye!
Hey there! Yes they would. You'll probably want to set up a 301 redirect.
Thank you Ellie, can you advice me a good article/post of how to do this, because I only know how to redirect from www to non-www and vice versa.
Konstantinos you dont need to set a 301 for this, just use rel="canonical"
I have a query.
I have xzy.com website and it sell residential film and commercial film. But the query is decorative film subcategory page is same in residential film category and commercial film category.
In this case I could not redirect with 301 so I block one page in robots.txt file.
Is this perfect way to avoid duplication issue?
Hello,
There are a few ways you can handle this. You can change your URL structure so that the category folders are omitted. For instance, I assume your problem right now is that you have two URLs for the same subcategory, www.xyz.com/residential-film/decorative-film and www.xyz.com/residential-film/decorative-film. If you omit the folder paths your new URL will always be www.xyz.com/decorative-film But I don't know if these is an easy change for you.
Another option is to add a rel=canonical tag to one of the URLs pointing to the other one, which will tell the search engines that they are the same page and to only index one.
You can also noindex one of the pages we well.
I wouldn't recommend blocking that page in robots.txt.
Hope this helps.
Hello,
Once they warned me that I had copied an article. When I saw that someone had copied my work, which cost me enough to do, what I did was talk politely with Mr. and removed. I understood that if criminalizing Google duplicate content, resulting in a worse position. Not so?
A greeting,
Hey there! Search engines don't actively penalize duplicate content, per se; they just don't always understand it as well, which can lead to a drop in rankings. Hope that clears things up!
I'd like to post a Q&A I have opened on this topic. https://moz.com/community/q/site-wide-content-like-...
It's about a technique many sites use: having the "why choose us" content just above the footer on every page!
Unfortunately, it's more frequent to see it than it should be. Indeed, some one has stated on my Q&A that she has on her company site this and it doesn't slightly affect the SEO. I mean, perhaps it's true that it doesn't affect your SEO, but there isn't any proof of evidence that without this site-wide content that site ranks higher, lower or stable than now.
Moreover, I am wondering how we can execute this in the proper way. I think it can help to covert, but only when it's not misread as the page content. For example, SJD accountancy has the biggest why choose us monster! Cast an eye on this site and you will find this content - over 450 words and 15 rich anchor texts to internal pages - that sticks at end of the article without anything in order to split the unique content with this.
You can see only a line and they marked it with <div id="content-bottom">. It also "carries" an H2 that pushes it up on topic hierarchy of this page from a clawer's point of view.
I hope some of you go to this site and give me a feedback, as I am wondering why this site ranks so well! They break 2 main SEO guidelines:
Some one says that Google has stopped putting a little weight on footers links and content, so SJD could rank well because Google doesn't read that content to extract SEO factor. But that's is coded as "content-bottom" not footer.
Thank you, your explanation of canonization was more to the point than Google's
Elien,
You come with a very good solution about the duplication issue. The main thing which I like most is, how we can use Moz Pro to identify duplicate content page and what steps we should have to follow to overcom from it. I think to add CANONICAL tag is the better solution. And I am pretty much sure that others will agree on it as well.
That's a nice post Ellie, you can also use robots.txt to block a certain directory and avoid duplication. For example if my website has a particular directory of specifications or descriptions in bulk (In this example /descriptions) and I cannot re-write all its content, I can simply block it through robots.txt, by adding this line.
disallow :/descriptions
Hi EllieWilkinson,
Thanks for the article, Need a clarification on some Duplicate Page Content Pages.
---> For Entertainment websites, most of the Duplicate Page Content form Search Pages and Gallery
---> For Search Pages we are showing the result for each and every page different but still it's showing Duplicate Page Content, Suggest me best solution for it.
---> For Example in the Photo gallery, we are differentiate each and every image in the gallery by change Title Tag, Description Tag, Image Name and H1 Tag even we are in Duplicate Page Content, Please suggest me.
Thanks.
I would add that duplicated content doesn't help not just Google but customers experience. If there's not a logical strategy for developing unique content, then the best is just to redirect 301. If we need all the pages (duplicated and very similar could be sometimes understood as synonyms), then rel=canonical solution is the best for me.
hi Ellie this is very nice post,
thanks for sharing the post
Hope guide me to be a good blogger. Newbie...