Addressing the Duplicate Content Issue

by Gary on September 24, 2010 · 3 comments

in Duplicate content,SEO

Time and again Google and the other search engines have explained the fact that they do understand that not all duplicate content is malicious. In fact, Google even expressly gives examples of non-malicious duplicate content (i.e. pages that are stripped-down for mobile devices but still essentially contain the same content and printer-only versions of web pages) in the Webmaster Tools Help page on the topic. Furthermore, they also say that snippets and quotes are not considered to be duplicate content, so that you do not even have to worry about being penalized SEO-wise for this.

Despite this fact, duplicate content is still a big issue in SEO, not for fear of being penalized but because Google might rank a URL of a web page with duplicate content higher than the preferred URL. The ramifications of Google showing the non-preferred link in its search result range from usability issues (i.e. users will go to the page with the stripped-down version) to monetization issues (i.e. all the Ads will probably on the preferred URL).

Several years ago, it was recommended that crawler access to duplicate content simply be blocked, usually using the robots.txt file. However, this is now actually discouraged. Google suggests that the best way to go is to when it comes to ensuring that they show the preferred URL in their SERPs is to simply tell them which your preferred URL this. This method is called canonicalization, and can be actually be done several ways. The preferred way nowadays, since all major search engines use this method, is to simply use the canonical tag by adding a <link> element with the attribute rel=”canonical” to the <head> section of non-canonical pages.

Another way to address duplicate content issue is to use a 301 redirect. Note though that with this method the user, and not just the bot, is redirected to the preferred URL. This won’t work if some users really need to get to the non-preferred URL.

Share This Post

marccclevy September 25, 2010 at 6:46 pm

Hi Gary, it’s been a while since I have written around here, but I thought i’d start contributing to this community again.

Definitely starting to see more and more SEO’s and websavvy people make use of the canonical tag, but I am also seeing a lot of people implementing it wrong! This can actually be very detrimental to a sites performance, so be sure to do your homework before implementing.

Gary September 26, 2010 at 10:02 am

Good to see you back Marc.

We are now going to be posting more regularly, and have a site update planned over the next few months.