It's late night Sunday and rather than bringing you exciting news from the world of search and SEO, I've got some explaining to do. For those who hadn't noticed, SEOmoz has some serious downtime and errors this weekend. Starting early morning Saturday and running through to Sunday, many parts of the site were inaccessible due to either A) an exceptionally unlucky set of simultaneous hardware/software failures on our host and backup servers, or B) the act of a vengeful Norse god (Odin, we're looking in your direction).
In any case, this catastrophe was exacerbated due to our recent hosting move - whenever you're shifting host locations, there's a certain amount of finger crossing to be done, particularly with relation to data backups. Jeff & Mel did a great job here, but this unlucky strike had a few casualties which couldn't be recovered.
- Most significant was the loss of the last two weeks of stored data from the Rank Tracker tool. We will again migrate all rankings saved before the Rank Tracker launch, but rankings and settings stored after the launch are not retrievable. We cannot apologize enough for this loss and have taken steps to ensure that this will not happen again.
- The Rank Tracker tool itself is still down while the migration completes, but should be back up in the next 24 hours. When the service returns, you will find significantly increased limits for manual rankings run per day and for automated rankings as well.
- A smaller data loss may be noted for saved Linkscape advanced reports. Some reports run in the last day or so may not have been saved. We have added 5 Linkscape credits to all accounts to accommodate this.
- Blog posts, user profiles, Q+A questions, marketplace profiles and any other database-reliant content created between Saturday afternoon (when the site briefly recovered) and Sunday midday (when our last round of failures/attacks occurred) is irreparably lost. Again, we're taking steps to ensure that backups will be secure and solid for the future, hopefully making this a one-time only event.
- Other fragments of data, including some Q+A questions, blog post & YOUmoz images, profile pictures and other database elements from the past 2 weeks may also have suffered. We've recovered nearly everything (and possibly got all of them), but if you notice some oddities, this is most likely why.
For PRO members, we're doing our best to make up this weekend's events to you with greater access to those tools hit hardest. We'll also work tirelessly this week (despite the SMX Advanced conference in Seattle) both to prevent future mishaps like this and to provide a high level of support for anything you need.
To our readers, visitors and regulars - I offer my personal apology. I know that you like to read and use SEOmoz on the weekends, and I'm sure this interfered with your regular course of business. We've got a great team of developers here at SEOmoz, and I'm incredibly proud of their performance under fire over the last 48 hours. Rest assured that all of us will put our shoulders to the wheel to make data integrity and uptime priorities over the weeks and months to come.
Thanks for your patience and understanding,
Rand Fishkin, CEO
p.s. Our thanks also to ex-mozzer Jane Copland, whose quick eye and catlike reflexes over IM brought the site issues to the attention of our dev team very early on and probably saved us additional heartache.
p.p.s. A few other areas are affected - Labs, Q+A Search & our RSS feed. We're working to get these back online today as well. If you find anything else, please post in the comments; we appreciate all the help!
UPDATE 2: We're going to go down for a couple hours around 1pm Pacific, 4pm Eastern today in order to secure our data in additional locations. We hope to be back up by 3-4pm Pacific this afternoon. You can follow the SEOmoz Twitter account for the latest on this front.
Early Saturday morning Internet browsing FTW! I was at work at 8am Saturday to let some electricians in (the fun of living closest to work!) and saw that SEOmoz was returning a 404. Ben was online, so I IMed him. Glad to have helped--suppose it could have gone unnoticed for a few more hours otherwise! I'm sure you guys didn't have a fun weekend as a result, but good to hear things are on the way back.
This is probably the first time anyone has apologized to me for their server being down. Really, Rand... I appreciate your dedication.
Well done on getting back what you have...! What a set of events. I hope you all get some time to relax at some point...
SEOmoz- Y'all's customer service and interaction with your members is UNREAL. I couldn't think of a better way to handle this "series of unfortunate events" Luckily I spent my weekend floating down a river drinking some beer, so I didn't need the moz.
Shiznit happens, keep up the great work!
RSS is down "Error creating feed file, please check write permissions."
Was wondering why my netvibes feed died...
I can live with a little downtime and the rare hiccup. I'm just glad that no bones were broken nor lives were lost.
Sorry to hear about al of the heartache! Great job tech support on keeping her afloat through the barrage! =]
Guys - I know ya'll are pulling your hair out. Me too as I have May reports due. Any idea when Rankchecker will be back online??
Breathe. Breathe deeply. When that doesn't work; drink heavily.
It would be interesting to hear if these tools get much higher use at/after Month end - maybe there's something to be said for changing your schedule to avoid the hordes.
I have lost all my linkscape reports...had over 10 saved reports and only getting 5 credits back!!! Plus the time having to re run them all again. Come on Rand dish those credits out a bit more.
Go here -> [email protected]
Really sorry about that - we can definitely help out via [email protected].
Still waiting on linkscape .edu advanced reports to be fixed.
Either they don't run at all and show a white screen that simply says "error" - mit, stanford, phoenix, uwm
or they return incorrect anchor text -
Reports 35272 and 35274 show garbage on the URL Anchor Text tab and on the other tabs that show links the anchor text doesn't even appear on those pages. 35272 is a bunch of German crap and 35274 is a bunch of weird forum usernames.
Ticket has been in for 2 weeks now.
Just logged in to Linkscape...all of our saved reports are missing...anyone else experiencing this?
very interesting post. Thanks a lot for sharing it.
Hi Rand,
Glad to hear you were able to sort out some of the challenges encountered these past few days. I'm an avid reader of Seomoz & I was able to see that it's returning 404, I thought that my dsl connection is going gaga, but when I pinged the site, it doesn't return any response. Anyways, you know what they say: Challenges/problems are blessings in disguise. Take Care guys! SeoMoz still rocks!
trifecta is down :(
My keywords need to be measured!! :)
I couldn't access the site several minutes ago.
Hopefully nothing was last this time.
Yep, I noticed (and reported) that Linkscape was acting up, and my toolbar has been acting funny as well. Something with the server? Frustrating, but I appreciated the email reply from your site support team explaining the situation.
I had to work all the weekend long and have accessed (or at least tried to access) seomoz.org many times in last 48 hours. First time (saturday early morning, US time) I've seen problems, I have thought it's temporary down (something like occasionally server restart), but after that I saw first data losses and knew it's a bit more serious.
You have lost some data two times. You have mentioned second, between Saturday afternoon and Sunday afternoon. But some data is gone on Saturday morning. I had a PM communication with someone from your staff and got an email notice that I got new PM on seomoz. This mail came at 12:53 AM (my local time, GMT+1) and that was probably very short before first crash. After recovering the server I have accessed my seomoz inbox, but the msg was not there.
On Sunday afternoon was the second crash and after recovery a lot of users' blog comments made on Saturday afternoon was not there.
I wish you good luck.
Thanks for the information. Hope, you'll get all the stuff back again soon. It's better to have the failure over the weekend than in the week (at least for us, not your team). :-)
Sounds like everything that could have gone wrong did go wrong. Hope you are past the worst of it now and get a chance to breath again. I can only imagine how stressful it would have been finding the cause of the problem and then discovering your parachute had holes in it.
Additionally, I just discovered that Q&A search function don't work.
For a term "keywords" I got
No search results... Try being more specific...
Was it an attack from Norse rather then a crash?
Spent most of saturday and sunday in panic of all the data I wanted to push through the ace tools, but no luck. I'm just glad you guys sorted it out. Nothing worse then complete serverfailiors and a big applaud for you server techies that fixed it. Make sure to buy them a beer (or coke).
not a beer, at least two beers. That was over the weekend activity.
Late Sunday night (UK) prepping my end of month client report, whilst keeping one eye on Steve Stricker as he sneaked a win at Colonial. I know, I thought, I'll spice up this report a little with a 'keyword difficulty' reference from The Moz: oh, oh ... no Moz ... no worries ... off to bye, byes ... finish report on Monday. Welcome back, nice mea culpa too!
I'm glad you got it sorted out. I was going through some serious withdrawl (twitching, sweating, etc).
At least it happened on the weekend, when your site usage is probably lower than during the week. :)
It looks like about 20 to 30 blog comments (user comments, not main topics), also went walkabout.
The last comment from before the downtime that is still visible is /blog/7-reasons-why-you-might-not-be-making-money-as-an-seo#jtc87097. This was a comment made on 2009-May-30 14:11 UTC / 2009-May-30 07:11 PDT.
My comments #jtc87110 and #jtc87111 made about 9 to 10 hours before you reset everything on May 31st, are no longer anywhere to be found. I am guessing that perhaps another dozen plus comments were made by other people after mine (there was a long period where the site was not accessible), and before the reset. They will also have disappeared.
After the site came back up, my reposted comment at /blog/the-web-developers-seo-cheat-sheet#jtc87099 made at 2009-May-31 23:08 UTC / 2009-May-31 16:08 PDT is re-using comment number #jtc87099 meaning that all comments from #jtc87099, up to #jtc87125 or more, and which were made on May 30th, and early on May 31st, before the downtime, have been lost.
Those numbers have subsequently been re-used for new comments that have been made after the site came back up.
I'm sure that armed with those comment ID numbers most people will be able to peek into their browser history and work out if their own posts were affected; and then re-post the information again.
I never did find comment #jtc87098 on the site, so I am not sure if that one was lost and re-used or whether it survived.
I pinged Rebecca and Jennita via Twitter several times on Saturday afternoon (UK time) and on Sunday when the site was playing up. It's lucky that just a few dozen posts went missing from the blog. Those will most likely be reposted by their author over the next few days.
I was once a member of a forum that irretrievably lost their last 6 months of user data. They never recovered from that; I never went back.
And I thought my posts this weekend were pulled for being a waste of space! Glad to see you made it through the series of craptastic events.
Hopefully you can repost them - if you can remember what you said, and where you said it... :)
Summer just broke in the PNW, SMX is in town, Oh man, "when it rains, it continues for months". Keep up the good work, thank you SEOMOZ techs! My weekend was spent in a blues joint hunched over an untouched glass of Chivas worrying about my favorite SEO company, it was hard. JK! Remember Nietzsche, that which doesn't kill you makes you stronger. Great job. Thanks for your hard work.