In February 2011, a client I was working with ran global usability test in cooperation with Spotless Interactive in London, Hamburg and Oslo. One year later and the feedback from the usability tests are being implemented, which includes improvements to the booking engine, navigation menu, a url rewrite and a new home page.
With a total of 21 websites in 15 languages, launching a new global web structure without losing a significant portion of traffic was always going to be a difficult task. Especially as more than 50% of total visits are from search engines. This blog post will show you two examples of how to launch a new website:
- Website 1 launches and loses 35% of organic traffic
- Website 2 launches and loses 4% organic traffic
By following the process described below, you can feel confident in knowing your new website launch will be smooth.
Minimizing risk and measures we took
With the support of Norwegian inbound marketer Sverre Bech-Sjøthun, we were able to create a step-by-step plan to implement before, during and after the launch.
Before the launch
We started by setting up a project in Basecamp with the web developers and to ensure all stakeholders were aligned.
Having just recently upgraded to the newest version available of the CMS, we found that the steps taken to minimize the traffic loss during the website launch would also be a good time to address to increased crawl errors reported since the upgrade.
By upgrading the CMS for more than twenty websites, the number of crawl increased tenfold to more than 4,500 errors globally.
Example of errors reported for one of the sites
404 page visits increased during the same period
The first step was to approach each site separately and by using Xenu Link Sleuth we are able to reduce the number of broken links. We repeated this process for each of the sites. Across the entire web structure, fixing broken links was a lot of work and requires a dedicated person. Understanding the importance of addressing these issues is the only way it can be prioritized above everyday tasks.
We then started working on the URL mapping:
- Using Open Site Explorer we ran a report for the top 500 linked to pages
- Using Google Webmaster Tools we ran a report for the top 10 Links to Your Site
- Using Microsoft Excel, we mapped out the site and new URL structure
Managing the top 500 linked to pages in a time consuming process but highly important when launching a new website. The process took half a day per website and was the most demanding of all steps taken.
Once the URLs were mapped, we then created an XML sitemap based on the live version of the website.
The step-by-step process for launching the new website included:
- Map URLs and redirects
- Submit XML sitemap
- Fix crawl errors in Webmaster Tools
- Monitor web traffic in Google Analytics
During the Launch
Once the new site went live, the 301 redirects were implemented and the XML sitemap submitted to Google Webmaster Tools. Traffic was monitored in Google Analytics and errors monitored in SEOmoz.
How to launch a new website redesign
The following illustrates how one team gave SEO a high priority and how another team didn’t. Here are the results.
Website 1
You will always run into problems that you did not see when launching a new website. The process plan was created and all stakeholders were aligned. However, as a team we did not execute the plan when launching this website and implementing correct redirects and on-site SEO were not prioritized.
The day we went live with website 1 (14th June, 2012):
- No XML sitemap was added to webmaster tools
- Not all 301 redirects were implemented (more than 50% missing)
- 302s pages were sending traffic to a soft 404 page (not a 404 HTTP status)
- Missing meta tags including page titles and meta descriptions causing duplicate content
One issue was that 50% of the redirects were not implemented. The issue being we did not know which 50%. Using SEO automatic bulk URL checker we manually checked each URL and HTTP status code. A second issue was that Google was indexing the test server resulting in duplicate content - more than 276 pages were indexed.
By not implementing the process plan, we lost a lot of organic traffic. In fact, organic brand traffic decreased by 45% compared with the previous week and year on year organic traffic was down by 49%.
Organic traffic is now down -34% comparing the previous month (easy to identify the launch date)
Impressions down -18% comparing the previous month
Website 1 is the client’s most visited website within the global web structure. The above charts from Google Analytics have been seen by senior management and addressing these issues has now been prioritized. It’s not too late, but there is no doubt that by losing 34% of organic traffic a considerable amount of sales have also been lost.
We are now in the process of updating XML sitemaps, implementing any outstanding redirects and fixing crawl errors on site.
Website 2
For website 2, we had a lot more control and I was allowed to be hands on with the process.
The day we went live with website 2 (26th June, 2012):
- Uploaded the XML sitemap immediately after launch
- Mapped out all URLs of the site, which included a URL rewrite with user-friendly URLs
- 301 redirects were implemented and tested
- Monitored the web traffic for both referrals and organic traffic
Once live, I blogged about the new launch, tweeted the launch to more than 2,000 followers, informed all stakeholders internally and had the news published on the company intranet. The day following the launch, we sent out a newsletter to 1,600 subscribers that included a tip to "check out the newly launched website!". The tip can also be found in my email signature.
Organic traffic is now down -4% comparing the previous month (barely visible)
Impression share trend continues as before
Here is a list of actions taken for the website 2 launch.
Actions summary:
- Create report of top 500 linked to pages from Open Site Explorer
- Map URLs from old site to new site with redirects
- When launching new site, implement redirects
- Submit XML sitemap to webmaster tools
- Test new top 20 linked to pages for correct 301 implementation
- Attract new site links through blogging and social media shares
- Send out newsletter and inform customer base
- Promote launch in company email signature
- Monitor traffic in google analytics
- Monitor and fix crawl errors in webmaster tools
- Submit new XML sitemap (two weeks post launch)
Concluding summary
With the website 1, we only followed the 50% of the plan and we lost 34% of organic traffic. With website 2, we followed the plan exactly as it should have been, we constantly monitored traffic in web analytics and tested both referral links and 301 redirects – a valuable lesson in having a plan and sticking to it.
If you are about to launch a new website, have your SEO consultant on-site or hire an expert during this process and involve your web developers throughout the launch. Make sure this is prioritized within the organization and not left to those who do not understand the importance of SEO. No one can afford to lose 34% organic traffic.
About Steven: Steven is an online marketer and works in luxury travel. He also blogs at tribes.no.
Having a 301 redirect guide is so important. Mapping them out on a one-to-one basis helps ensure that everything is all lined up BEFORE you launch the new site. I too have worked with a client that didn't have their 301 redirects in order before they went live and they lost about 30% or so of their visitors. It's one of the most common mistakes I see sites make when undergoing a redesign.
Steven, valuable advice, especially now that every link really counts. It's amazing how much traffic can be lost without a good migration plan. You might like our method for a scalable way of redirecting historic links discovered with Webmaster Tools. It uses Google Docs and a little scripting to quickly provide redirects in bulk for an .htaccess file. https://webbactivemedia.net/marketing-blog/289-post-penguin-bring-back-the-lost-link-love.html
Thank you for sharing this valuable resource Jeremy, this will no doubt come in handy in the future
Great article Steven. I think it is imperative to involve SEO when changing so much (especially with larger sites). When a business is relying so much on organic traffic, losing up to 40% of traffic can be crippling. My company has a much smaller site that we are redesigning and launching by the beginning of the year and this post definitely reminds me to stay focused on how to make sure we do not lose our organic traffic. Thankfully, we are an SEO and Web Design firm so it is fairly easy to make sure everyone is one the same page
Awesome, being on the same page is key
This is very interesting as I happen to be hired just after a scenario worse than the 1st scenario you have described.
I know the pain. The traffic loss in my case was more than 50% as a majority of the URLs were not redirected.
It has taken me around 7 months to get close to the traffic levels before revamp.
What would be your plan if you had a set of over 10 million to deal with?
Sounds like a tough job, but glad to hear you are finally getting traffic back to where the levels once were. Is that 10 million URL's you had to redirect?
yes, 10 million. Ran out of ideas how to do it. Hired some contractors, but didnt help much as the old website was gone....
Saibose - I'm Sverre, Stevens helping hand in the move. Though hard, with the old structure "gone" and all, it's entirely possible to reconstruct the old structure using Web analysis and link profiling.
While you may not be able to get all 10 mill URLs, you should definitely be able to pick up your most important URLs, thus salvaging the majority of link juice to your site.
Let me know if you want to discuss things further :-)
Sverre
I would suggest you hire some contractors to help you map your URL's at that level. Use Odesk or something similar. You will still need to go through them all with a fine comb but it should speed things up a lot.
Interesting thoughts and I highly agree with the fact that one should have to have a plan before the site should move from old version to the new version. Recently a client of mine changed the complete website that includes everything from server to URLs, design and other stuff all at once and as a result he left almost 70% of the organic traffic and a serious dip in rankings on highly trafficked key phrases.
I believe the planning is necessary or else things can go really crazy and damage can be permanent which need a fresh start to get the traffic back from search engines.
Great call Steven!
Thanks, appreciate it.
Thanks for the sharing. good one..
If we apply 301 redirect for about 70 URL(internal links) with new URL link restructure, which are having good position SERP presently. will there will be an drop in position in SERP or ranking of site..??
Thanks Steven - I'm going to be doing a couple of site re-designs over the next couple of months - so this was a VERY timely article for me.
I am also curious about bhsaiuday's question above along with, does there ever come a time when you can remove the 301 redirects from the htaccess file, or do they need to stay in there forever?
Speaking personally I didn't have any problems when I 301 redirected around 100+ URLs from my website. As long as you update your XML sitemaps / Feeds you should be fine - just make sure you resubmit them to Google Webmaster tools.
I'm totally agree with 301 redirects and the sitemap submit. If any one need to launch a new web site please be consider those,, otherwise it'll be a big matter for your total business.
I have been through the site/cms/redesign migration process several times in the last 2 years and we haven't lost traffic not once, in fact we gained every time. What we believe was the reason for traffic gain is:
- every time we took the opportunity to review and re-work IA of the site, with the emphasis on traffic driving subjects and pages. We decided whether some pages needed to merged, whether there is a page that should be split in two pages, whether there is a gap in the content in order that the site is well-rounded on the subject it portrays etc.
- identifiable content gaps were filled in and IA adjusted
- there were no mistakes made during migration launch process (redirects were there, there were no influxes of multiple URLs from the "new" structure for the same page, HTTP response codes were correct etc.
- we reviewed (and changed where required) all page titles / descriptions
As Steven, we have analysed current site URLs to identify:
- which pages to redirect, and where
- which pages to let go 404
To identify current site URLs, we used:
- Screaming Frog spider (great interface)
- Google Analytics landing pages from roughly one year back
- Google Webmaster Tools current site crawl errors section
- Google Webmaster Tools duplicate titles / descriptions section
- Google Webmaster Tools Search Queries landing pages
- site: operator in Google search
We then also checked external links to current site pages.
This usually resulted in a long spreadsheets which was then filtered down to unique URLs and an action whether to redirect or leave it go 404. This and the new IA/content gap analysis were the most time consuming parts of the process.
When the new site was ready (in test environment, blocked from robots), we then used again Screaming Frog to test current site URLs and also to test redirects (importing known old site URLs as a list to Screaming Frog and let Screaming Frog Spider run of that list - which should produced either 301 or 404 for each URL on the list).
Once the new site is ready (in test environment), the testing period is very important as here you should pick up any redirect errors or URLs that may cause duplicate content.
The sitemap.xml that we produced had always the new URLs - we never left "old sitemap.xml". We believe this gives better technical quality signal to Google and in any case, Google does not need to be reminded what your "old" URLs are - it keeps the long history on these. In any case, with the influx of new URLs, Google will be busy crawling new URLs on the site
(BTW, Google throws error in WMT if you have an url in your sitemap.xml that redirects or is blocked by robots.txt)
You also need to know that for a while new pages will be seen as "duplicates" of old pages - until the old pages are re-crawled and 301 processed by Google. This *may* result with a temporary duplicate content, but should rectify itself soon.
Then after moving the site from test environment live, there is a period of few months (at least) to monitor URLs and rectify any errors reported.
For anyone who will be going through migration process - it needs an attention to detail and following a plan. Do not let yourself be rushed by the client saying "it must go live tomorrow, lets put out whatever is there" as the recovery is always longer than getting it right from the beginning.
Apologies for a longish post!
Without a plan is like you are sailing without compass.
Great quote Nik
You are just reading my mind ;)) Want to write exactly same...
301 is really nice solution and to submit XML sitemap that is really helpful.I just had a case of my friend who lost his website traffic by 50% after launching new site was facing problem of links those were not redirected.I like the topics to plan before launching new site.Thanks
Great article. As I always tell people SEO begins before a website is even published.
It's great to see your project and how you launched a website from beginning to end. Keep up the good work!
Appreciate the kind words, thanks!
Great analysis of the 2 different projects. Thanks.
Hi Steven, I have read your full post. This is just mind blowing. This is very helpful for all. I am founding this kind of to be a seo expert Sydney
[link removed]
Thanks for the positive comment
Steven, Thank for Sharing such a valuable and informative Post . Every webmaster has need to reconstruct the website and of course by this tips we can maintain our website traffic.
Thanks
Hi Steven,
Your study is excellent but recently i redesign my site then it is surprising changes in my ranking and traffic also, i got new amount of visitors.
Very cool!
What is it you did to increase traffic?
Yes, i show tremendous change in traffic and bounce rate.
Thanks for this great share. I've found Screaming Frog to be great (as is Xenu Link Sleuth) to do this type of restructuring. Could you give a little info on the type of 404 page you used for website 1? Did they, for example, contain many links or use a standard error text?
Hi,
Sure, no problem. The 404 page used was very basic in being a huge image with only a link to the home page. Another problem was that it was reporting back as a HTTP status 200, and not a 404.
We have since improved the 404 page to include a navigational menu to categories on the site as well as providing an email address to those who wish to contact us further,
Thanks for sharing. Nice to see real project stuff like this and a useful checklist too!
I hadn't seen that bulk URL checker before. Really useful. I'd previously done this through SEO Tools for Excel (https://nielsbosma.se/projects/seotools/) - but that can be really slow for large lists, so thanks for that link.
Thanks Matt,
I thought I'd include the checklist just to make sure if anyone else is involved in a new website launch, it might be good to see the steps we took.
No problem on the link to SEO Bulk Checker, since finding it it has been incredibly useful (and as you say, timesaver!).
Having a plan is extremely important. Having just gone through the same process it was important to do research before the launch and develop a plan to minimize negative impact. Things always come up that are not expected but the plan was a good point from which to address the other impacts
Great advice, redirecting specific URLs to the same or similar content on a new site is definitely beneficial. Often sites will just throw in a site-wide redirect to the new homepage, which does not help search bots index the new content / credit the existing link juice and is also a poor experience for users looking for specific content.
Many thanks for this post! I'm working on a redesign and had to convince the web developer that the 301 redirects are worth the effort - she was going to have everything default to a 404 page. Will save this article to help the next time I need to explain it.
Loved the Idea about infographics. Certainly helpful!
Hi Steven, your post is around the web with new URL structure but if someone launching new web design without changing URL structure what would be your suggestion to look and care about?
Hi Syed,
My recommendations would be to follow the same steps without the url rewrite. I would also suggest running your beta site through www.usertesting.com before launching to identify any minor user issues before going live.
Thanks
Steven,
Great learning opportunity there.
You mentioned that you redirected the top 500 URLs. What percentage of total URLs is that, or what determined that number? Was your URL list including only those URLs with up to x inbound links?When do you stop monitoring the server status codes?
Also, If your situation allowed it, did you consider using Regular Expressions or pattern matching to redirect a lot more URLs than 500? Might this be where your 4% was lost - URLs that were not redirected? Or maybe it is just a built in degradation in the search algos.
Excellent to see what a difference having and following a proper plan makes in relation to a site restructure. As the others have said it's always good to see two real world examples and have them next to each other for comparison!
I hadn't come across the bulk URL checker either, but I will be giving it a whirl on a current project. Thanks
You're welcome Matt, thanks for the comment.
The bulk URL checker has a limit of 50 URL's but it's still a very good tool to use.
Refreshing to see such good content on SEOmoz again. Fantastic article.
Thanks Daniel, really appreciate your comment
Great article and great advice - will have to re-read it to make sure it all sinks in.
Thanks.
Great Post we are facing the bounce rate issue these days.
Thanks for the tips. We have 42 websites and several of them are being reconstructed as I write this missive. A few of them are still in the recovery from Google's updates. Grrrr.
Love your post, excellent advice Steven!
Love the bulk URL checker - thanks for that!
Really useful post of things to check. Currently working through some restructures and very conscious of losing traffic - especially if it's through things I haven't done.
Hi,Great stuff dude. But I have one question here as I have one website on Scenic Landscaping. Everything is perfect both on page and Link building. Now from last 3 or 4 month the traffic decreasing gradually. So, can you please suggest some way to get the same traffic as it was 3 months before.
And apart from Webmaster tool, is there any option to Fix crawl errors. Any help will be appreciated.Thanks,Shaikh Zafar
Zafarshaikh92, Hard to tell with out knowing the URL. I suggest posting the question in the SEOmoz Q&A forum.
Hi Steven, thanks for sharing your story!
Your Actions summary list is useful as a checklist. Is this the exact checklist you used?
It would be nice to see this in infographic form complete with definitions/reasons. Then it could be shared with and be used by other SEOs who are "looking forward" to a client site conversion of their own.
Hi George, great idea on the infographic! I'll look into it.
Yes, correct. This is the exact checklist used. Thanks
You mentioned 404 page design. It's something I'm afraid I haven't given the issues as much thought as I should have.
Could I trouble you for link to take a look at one of your examples?
Hi Jeff,
I have just sent you a private message with the 404 page.
Thanks
Steven
This is a great post and will be really handy when presenting to the stakeholders what level of time and effort is required to monitor and implement a restructuring for our site. Just a quick question though. How did you use Xenu Link Sleuth to reduce the number of broken links?
Hi Sean,
I used the "Broken links, ordered by link" report in Xenu to fix them
Thanks for commenting
Hey Steven. Thanks for sharing this article. Really like the comparison of the 2 projects. There are still too many webprojects launched/relaunched without envolving a SEO. I'll def. Share your article at work later.
Thanks again and cheers from Germany Sven
Hey Sven,
Yeah, I agree! It's strange to watch projects go live knowing that without following the proper planning, they will lose both traffic and sales.
I really agree with Steven, I already observe this in some of site that I manage.. By just going with the quality on-page and proper off-page all of this will be fine.
How to do it? read more.. ^_^..
Thanks Steven for your documented share.
Thanks Bryan
Hey Steve,
The post is thoroughly insightful.Its very useful especially for novice bloggers and website developers. Organic traffic leads to link building that is essential for the blog or the website to survive.
Regards
Sustainable-Sphere
Appreciate the comment, thanks
Hi Steven,
Thanks for posting this article. I am just about to re-design a website for a client who gets a ton of organic search traffic but has not updated their website in 10 years. I want to make sure it is a relatively seamless transition with a minimal loss in traffic. I appreciate the time you took to detail each step and overall process. This will prove invaluable for my project.
Thanks again!
Brock
Best of luck Brock, and feel free to let us know how it goes once your new site is live
IMO linking from SEOMoz to alltopparadise.com with exact match anchor text ("luxury holidays") isn't a very good idea.
Not only Google has devalued exact match anchor text but more importantly your site can lose positions as posts from SEO Moz get scraped several times. That means that in a few weeks time there will be several "luxury holidays" anchors pointing to alltopparadise.com. Even though you would be able to update the anchor on SEO Moz, that wouldn't be the case for the sites with the scraped content.
Hi Steven, what about backlinks analysis ? I mean, I think it is important to monitor the most important backlinks your old site has, in case they point to sub-pages whose urls got changed when the new site went online.... so these backlinks could probably return some 404 errors.
Hi Niguli,
Great point, and was covered in steps 1 and 2. We also used data from webmaster tools and made sure our most important pages were redirected. Once the site went live, we monitored the site to fix any newly found 404 pages. Thanks
Great post Steve, I would really appreciate the points you mentioned in this post. Before creating the new version of a website we must have a proper plan to launch it so that we can retain our old website traffic on our new website.
301 redirect I think the best way to retain the traffic of a website and I it is the successful way to retain website traffic.
Great post, thanks - backs up why SEO should be an integral part of any redesign. I also found a new useful tool - the bulk URL checker :D
Thanks!
Hope you get as much value from it as I did.
Some extremely useful suggestions - I undertook a similar project to the back-end of last year. Whilst I knew what I was doing at the time it still would have been nice to have seen this article first - kind of reaffirms your knowledge a little.
Thanks for the positive comment
Hi Steven,
Great post, we recently (June) moved sites and used pretty much the same tools and plan as you mentioned above. We saw a drop in organic traffic of 4.13%, which was more than recovered the following month.
One element where I differed, was on the XML site map, in that I ran the old XML site map for a week after the new site was launched, to give the bots a chance to crawl the 301's. From seeing your results I'm not sure if this made much difference, which is why this post is interesting to see. Thanks Matt
Hi Matt,
I should have made it more clear but the approach we took was the same as you mentioned. Once the new site went live, we submitted the old XML sitemap just as you did. Interesting how we both only lost 4% of organic traffic (and both recovered the following month.
There must be a secret step we are both missing to go from losing 4% to 0%
Steven
Would be great to know this secret step... If there is one, I'm sure someone on here knows it!