UPDATE: 9/7/2012 4:00pm Pacific
We have fixed our rankings service issues-
- We are collecting today's rankings as planned.
- The Keyword Difficulty Tool is back up and running.
- We are backfilling the previous 5 days worth of data completely(Sunday through Thursday), but will be missing rankings from Friday the 31st or Saturday the first. Today's data will be complete by the end of the day, and the backfill will be complete by Monday, the 10th.
- Rank Tracker is still not functional, but we are working hard to fix it.
By the way, we are hiring, if you or someone you know would like to help develop our products, drop us a line.
Over the last 3 weeks, you may have noticed some instability with our Rankings tools through missing data and error messages stating some tools are unavailable. On Friday, we experienced a totally different, unrelated problem with our rankings data. We expect to have an updated prognosis for that problem by tomorrow, but we want to fill you in on what went down at Mozplex to cause these issues in the first place. To be as transparent as possible about what happened and how we're working to fix the issue, below is a summary of what was impacted, the work we did to get things going again, and what we’re doing in the future to make the system more resilient.
Database issues? What gives?
Our SERP data subsystem (which runs on the distributed storage technology Riak) had a couple of nodes fail. To learn more about Riak, here's a blog post we wrote when we made the switch last year. The subsystem is designed to handle such failures; however, we did not handle the failure correctly.
In the process of fixing our Riak storage, we disrupted some of our queues for SERP data processing. Given Moz's growth over the last six months and the number of SERPs processed in the Riak cluster, Roger can no longer recover from outages in a timely manner. In late 2011, we could recover the system in 3-8 hours and be caught up on data processing in a few days. This time around, it took us six days to get the system back up and another two weeks to catch up on the missing data and the inconsistent data states that resulted.
Impacted services
Riak stores our SERP data (rankings data), so all the systems that depend on it were impacted. The impacted systems include:
- Custom reports
- On-page reports
- Historical rankings CSVs
- Rankings
- Keyword Difficulty & Full SERP Analysis reports
Work completed to get things going again
Our dev teams have been hard at work to restore all missing and inconsistent data post Riak malfunction. At a high-level, here's what we did to get Rankings and all its dependencies going again:
- Created scripts to heal the different broken states of jobs
- Added more nodes to speed up processing and help in future failures
- Improved monitoring to get information about failures and performance bottlenecks
- Improved performance in a multiple areas
Future work
It took the team 20 days to fully recover from the cascading problems that resulted from the original issue. We know that this timeframe is highly unacceptable, and we apologize for not being able to recover quicker. We are now in the process of ensuring that the same failures do not occur in the future and to lessen downtime in the event something like this does happen again. Work has begun on multiple improvements to help us reach our goals, including:
- Improving health checks and threshold monitoring of Riak nodes and subsystem dependencies
- Adding more Riak nodes
- Beefing up queue and job execution monitoring and alarming
- Creating a dependency matrix that indicates what’s impacted when something goes down
- Improving fault tolerance in parts of the system
- Providing additional excess service capacity
- Creating system operations documentation for dealing with emergency scenarios and how to recover
So, what's the current ETA?
Unfortunately, as you can probably tell, we have a lot of work to do to get Rankings back to 100%. We don't have an ETA quite yet. However, we hope to have a solid date in place by tomorrow and will update the post as soon as we know. Again, we apologize for the failure and any issues it has caused. We are working our butts off to ensure it doesn’t happen again!
If you need an immediate alternative for rank-checking, try using the Rank Checker at SEOBook.
For status updates on this issue, please check out our Rankings page on the Help Hub.
While I appreciate the update and explanation, I for one am not satisfied with this response
1) The Keyword Difficulty Tool is the primary reason I, and I suspect, many others pay for a pro subscription. This tool has never been reliable and has suffered frequent downtime for well over a year.
2) If you are genuinely apologetic for the lack of service, why not offer pro subscribers a rebate for a service that they have paid for and cannot use? I'm sure you have a disclaimer buried somewhere in your terms and conditions, but the fact is that you are taking my money and not providing a service in return. I don't care how cool and transparent you think you are, that just stinks
3) If the tools don't work then why not display a message saying so or take the tool down until it does work? By allowing users access to a tool that doesn't work, you are just wasting everyone's time running queries that produce no results. At the very least, tell me there is a problem BEFORE I run my query.
All 3 points are so blindingly obvious that you couldn't possible pretend that you are not aware of these issues.
Do the right thing and I'll be the first to give praise
I second point 2 - my daily rank tracking provider has been having trouble recently too and they've thrown in a month of free service for the inconvenience.
That being said I still appreciate the TAGFEE.
Hey Dan - totally empathize and will try to address:
1) KW Difficulty is "sort of" working and we're getting it to fully working very soon. Hence, the error messaging is kinda tough, because we don't actually know on any given run if it will work (about 50% of requests currently function, but it's inconsistent). If/when we go fully down, we'll definitely change messaging to reflect that. And I'll try to work with the team to figure out something more consistent for messaging when things are inconsistent like they are now.
2) 80%+ of the PRO services are working (toolbar, API, OSE, PRO web app minus some rankings, social analytics, Followerwonk, etc). We don't have the ability set up to do partial refunds, and our service isn't currently billed or set up that way (we know we won't have 100% uptime, and we try to make that clear in TOS). We could do better with messaging and I'll work on that - it's a good suggestion.
3) I hear you - like I said, when it's totally down, we do this, but with inconsistent issues, it's tougher. We'll try harder on messaging more clearly for this.
1) Sorry, but KW Difficulty is not "sort of working". I have been attempting to use it regularly for the past 4 days. Only once did it produce a result. My experience is a failure rate of around 85%-90%.
2) Telling me that 80% of other services are working is like saying that I shouldn't complain about my coffee machine failing to make coffee because it will still froth the milk! If my primary use of SEOmoz is tools that rely on ranking data then I don't much care about your other tools that do work. Quite sure I'm not alone on this one, although possibly the only one with enough balls to voice an opinion.
3) You not being set up to offer partial refunds is not a justification to take my money and not provide me with a service in return. Issuing partial refunds is your problem, don't make it mine. If you needed to accept "partial payments", you would soon find a solution.
I appreciate that the problems may be intermittent and unpredictable, then say so. How hard would it be to add a message "Hey guys, We are experiencing problems and this tool may not work 90% of the time"? Show this message before I hit the submit button and you will be going some way to really honouring your transparency commitment.
I spend most of my days offering SEO training to large groups of small businesses. As part of my workshop I demonstrate SEOMoz tools to groups of your potential new subscribers. I don't take affiliate payments from you for this, I do it because I believe in your service. You are starting to make me look like a putz!
I completely understand your frustration Dan. I know what its like to have just a part of a service down, but it happens to be the only part I use. We are definitely working our butts off here to get this back up and aren't taking any of the gritty details lightly. Again, I know exactly where you are coming from and can sympathize. We really appreciate all the feedback too. I would also like to apologize to you (everyone else also) for adversely affecting your business.
I am going to send you an email and so we can talk about your account with a little more detail in a customer service capacity.
Totally hear you. We're not doing well enough and we need to get better, both on service and on ability to manage billing at a more granular level.
FYI - 80% of the time I've tried to use OSE this week I got an error. It's not working...
I understand things do happen, but before this whole debacle OSE was out of date for several months too.
Love SEOmoz though so I won't be unsubscribing. Hopefully all that new money you recently received will help speed things up.
Hmm... We had some downtime, but 80% this week sounds weird. Looking at our logs, something doesn't match up. Can you email [email protected] and they can take a look at what's going on?
Re: new money - yeah, we're building that new datacenter off Amazon so these damn EC2/S3 failures don't bring us down so much. A few more months of labor from all the new engineers/tech ops folks we've brought onto the team, but it's coming.
I don't mean to trivialise this post for more than a second, but there has been a lost meme opportunity that i would feel bad if i didn't rectify.
Where Are My Rankings? [fixed] https://i.imgur.com/1XD8h.jpg
Im glad you rectified it. LOL
Thanks for the transparency guys! FYI, this is also why I use a back-up system for rank checking too. This isn't because I don't trust Moz, it's because you can never be too safe in today's technological world, which with all its beauty and good parts also has its technical malfunctions.
Keep up the good work Moz engineering team!
Well - I am not as lucky -
What other services do you use?
If they are paid - it will really stink as I already pay a healthy amount for MOZtech
your pal Chenzo
Well, I do use a couple other paid solutions because I have access through the agency I work for, but you can also get rankings in Google Docs (and then if you mash them up against Analytics data in Gdocs, you can see your areas of opportunity).
This post on Distilled from a while ago gives away a spreadsheet for doing this:
https://www.distilled.net/blog/seo/how-to-build-agile-seo-tools-using-google-docs/
Hey Chenzo - Check out MicrositeMasters.com they have a decent tool, it's pretty much only rank tracking but it's a relatively reliable back-up.
We're gonna try to get to a point where you don't need to do that, but it's a fair suggestion given our imperfect data collection ability today. Thanks John.
I'm just saying that I like to have data from multiple places because I like to be able to compare it and make sure my data is as accurate and complete as possible. It's not a knock against Moz at all; I'm just distrustful of every tool and like to double check the data/results myself.
Moz is my go-to for rankings, especially with the weekly emails, but a backup solution (even a Google Doc hacked to track rankings) is never a bad idea.
Agreed, I use multiple link count tools (OSE, GWT, Majestic) for the same reasons.I don't like to rely on just one source, also multiple can give you a more complete picture.
Ladies and Gents, this is how customer service should be handled! Anyone know of any moz articles on the topic? My search proved fruitless.
How about sending out an email to Pro Members when the data becomes available?
Great idea irube! Unfortunately, each member's rankings will be updated at a different time depending where their campaign is located in the queue, so an email to all PRO members would be hard to time.
However, you can make sure you receive a weekly ranking notification via email from your campaign settings. If you've turned off these notifications, you can turn them on again to be notified when your rankings are updated (hopefully soon)!
You can find info about opting into campaign notifications here: https://www.seomoz.org/help/campaign-settings
UPDATE: 9/7/2012 4:00pm Pacific
We have fixed our rankings service issues-
The Keyword Difficulty Tool is not working.
Still seing this: Due to a server failure, we are experiencing a delay in SERP Analysis results this week. Unfortunately, it may take up to a week to get you back to Keyword Difficulty paradise. Thanks for your patience and understanding; our engineers are working around the clock to get this issue fixed. - Updated September 4th
Seems like I picked my 30 day trial at the wrong time.
Oh well maybe try signing up in a few years when I see you guys are really stable. I see over the past year you guys have had issues.
Thanks for the update guys! Good call on letting people know as opposed to just maintaining a poorly run product like some rank checkers I know of.
I'm also getting the error, and I reached out to the @SEOmoz Twitter about it, and they showed me this URL. The error shows up, if you run the query again, it will go through, but the first time it bounces back to the tool itself and the "this tool is down and unfortunately it'll take a week" error pops up.
Same on the Rank Tracker tool, "Due to a server failure, we are experiencing a delay in Rank Tracker results this week. Unfortunately, it may take up to a week to get it working properly again. Thanks for your patience and understanding; our engineers are working around the clock to get this issue fixed. - Updated September 5th."
I click all sorts of stuff on the Ranking Tool and nothing happens. Not how I wanted to kick start my week, I'll be honest.
I realize it is a completely unrelated support team, etc, BUT it is of little comfort that freaking FollowerWonk is working just fine (which I honestly haven't been convinced of the value yet) when the tools we've had with you guys for ages aren't working and I'm onboarding a new project today.
It's an irrational annoyance as you didn't build that (snicker) but still.
Is there a discount on our memberships? This feature is one of the main reasons I user SEOMoz
From what I see SEOMoz has never provided a refund or discount due to issues on there end.
Maybe I am wrong. The stability is one of the issues that's putting me off keeping my trial going.
The last company I worked for about over a year used SEOmoz but the web app was unstable at this time.
Hi hostsurf,
I'm sorry to hear that! We do our very best to keep our tools up and running and our Customer Service team is actually quite generous. :) I'd love for you to give us a chance to check it out personally!
Thanks,
Jen
If your customer services team was truly being "generous", you would be automatically compensating all pro users for a service that consistently fails to work. Instead you ask people to contact you privately and refund only those that shout the loudest.
I thought SEOmoz was supposed to be better than this? What happened to to the TA in TAGFEE? You may need to change this to FEG!
Hi Mark! Sorry about the slow response. Please shoot us an email at help[at]seomoz.org with your account details and we can chat a bit more. Thanks!
Thanks for the transparency, and keep up the good work everyone!
Miss all of you! :)
I was wondering... Heh. Thanks for the update-o-roo
Thanks for owning this issue guys. The mea culpa above is exactly what your members are looking for to cool their frustration with issues like this. I appreciate the open and transparent communication!
I definitely think you need to have a status bar or button on your tool as soon as it's down, maybe a green/yellow/red light. Green=100% Yellow=It's slow due to high number of requests Red=It's not working properly, in the mean time here's an alternative for you. It should show up on this page https://www.seomoz.org/tools and also on each of the tool pages. Your Google Adwords API is currently broken in your Keyword Difficulty Tool, I should know right when I hit this page https://pro.seomoz.org/tools/keyword-difficulty.
That said, thanks for the update
Appreciate it for that update best service result here! Very good call on allowing people understand compared to only preserving a inadequately function solution like several list checkers. I'm sure associated with it.
Thanks for the update Thomas! Although I was out of the total SEO scene and was not checking the tool so I am not sure how painful it was but I am sure my team will be tweeting and asking for help regarding this.
All I have to say here is that the reason why I have SEOmoz’s logo under the heading ‘My Love’ in a side bar is due to the level of transparency that you people have created over the years with your community and SEO industry in general.
Thanks for being transparent, this is some great stuff here!
Thumbs up for your honesty - it's very much appreciated and respected.
I needed something quickly because marketing came and told me they were investing heavily in a new campaign (the day before launch) and oh, by the way, can I generate reports to track the effectiveness (there's always lack of proper planning with these guys)?!
I heard that SEOmoz is one of the best tools and came here and then read that you were having some down time, so I tried the SEObook tools you recommended, but they're kinda lightweight. In the meantime I'm trying Rank Ranger and so far so good. But I'm open-minded and willing to give SEOmoz a try when you're back up running full speed ahead. I hope that will be soon.
Thanks for the update, it's appreciated, I wish more companies had the same policy about downtime/malfunctioning services.
Thanks a bunch for letting us know what’s going on. Too many companies lie or stretch the truth about what’s going on and say stuff like "It will be fixed tomorrow" repeatedly in fear of what customers will think.
I applauded seomoz for taking the time to let us all know the issues and what you’re doing to fix it. This is one of the many reasons I absolutely love seomoz / staff.
Kudos to you guys
Thank you for an update, I was wondering why I can see rankings from about 2 weeks ago and not any newer... I hope you sort out this issue quickly, so we do not have this problem next month.
We will definitely have it fixed before next month! We're all crossing our fingers it can be sorted by the weekend.
There is no such thing as 100% up-time when it comes to applications or website. We should all be grateful for every thing Moz is trying to achieve. The transparency model works quite well.
I'm sure some people still will be upset.
But the rest of us appreciate the honesty. I've dealt with too many companies who have had their fiber optic line cut by digging from an outside company every couple of months, while they fumble through fixing the real issue.
This is very helpful for me Thomas! Thanks!
All I can say is we are with you Rand :) .. It is a big issue but I am sure mozzers will do their best ....
Can you let us know when the api will be working again for url metrics?
Hey there!
Everything is back up and running with the API url-metrics call since Thursday afternoon - what kind of problems are you seeing? Feel free to shoot me an email at [email protected] with the information: member key, API query, any errors you're seeing.
Thanks!
Carin
My mistake. It looks like url metrics is fine. What is the status of the Link call?
The links call should be fine as well - what kind of issues are you seeing with the Mozscape API Links? Feel free to email me if you are still seeing problems!
Thanks,
Carin
[email protected]
Nice Sharing Thomas, very helpful
Uptimes and downtimes are common in business. What we you got to do is to give the excellent customer support. Hope SeoMoz fixes it soon.
Still getting the same error with Keyword Difficulty tool when tested 5 mins ago!
Thanks Thomas and Rand for letting me know what's going on, I appreciate it's quite a big issue, nodes blowing up left right and center. Sounds like something out of Natural Selection! :D
Glad to see what your planning in the future to avoid them happening again.
Can I get a free hug from Rodger as an apology? ;) haha
P.S. richdan, relax bro. He said they're hoping to have it up by the weekend. Plus the froths the best bit! :)
Roger is totally ready for all and every hug. =D
>> P.S. richdan, relax bro. He said they're hoping to have it up by the weekend. Plus the froths the best bit! :)
I wish I had richdan as my customer.
Complaining (especially on this site) is a HUGE risk. And his honesty is a breath of fresh air.
Not only would I give him a mind-blowing credit (probably 3 months at the least), I'd stalk him down and try to talk to him on the phone personally. I'd thank him up and down for his feedback.
Then I'd ask if it was okay to setup a meeting with my engineers and offer to do a conference call with richdan at the same time. I'd pick his brain on how to make things better... blow by blow - I'm certain richdan would be thrilled to help and his feedback would be worth more than gold.
Here's the best part: once the issues are resolved, every wins - I'd bet richdan would provide a followup comment praising SEOMoz for turning a bad situation into a win.
Those are some great ideas Markus! I have already reached out to him. Thanks for the feedback on getting feedback. :)
Thanks Nick I would be happy to hop on the phone with you. We are always open to feedback and getting better.
I think "Nick" probably sits 10 feet from you. No need for a call lol.
I bet Anthony wishes he sat that close to me! He is sardined into a room with the other execs. Never seen four execs share a office smaller than a closet. It really should be a sitcom.
It's a pretty sweet set up and yes I do need to spend more time with you all. It is one of my goals!
Markus, you would win that bet! :)
All very good points Markus :) Thumbs up.