Index update 5/9/12 (from Carin)
The full index is now live! This index is exactly the same data as what was released on 5/1, but includes the updated Anchor Text views.
Index update 5/1/12 (from Jen)
Hooray! The new index went live today and it's BIG. Check out the latest blog post for all the information. We all thank you for your patience and support (and understand your frustrations).
Index update 4/24/12 (from Jen)
Unfortunately we ran into a hiccup over the weekend and will not be able to hit the projected 4/27 date. The team has been working like mad to get everything resolved and the next index is scheduled to launch between 4/30 and 5/5. During this time we'll launch the links and url metrics first, with the anchor text shortly after. We'll update as these go live (be sure to watch Twitter for updates). The good news is that the index will be 3x the size of the previous index.
We know this has been hard on everyone and we really appreciate all your patience and feedback. We'll keep you updated as we progress.
Last night we had to delay the Linkscape index launch date yet again to April 27th as can be seen on our release schedule. This is the second time we have had to adjust the launch date this processing cycle meaning there will be almost two months between index releases. We know this has a major impact on the SEOmoz community, so we felt like we owed you all a detailed explanation.
Back in December, Rand and the Linkscape team decided to ambitiously try to grow the index to over 100 billion URLs. Such an index would increase our coverage of many of the smaller sites on the web and also help us shape the breadth-versus-depth trade off in our crawl. We're very excited about this prospect, though the growing pains have proven to be a challenge which required us to tackle several issues all at once. In the interest of transparency, we want to fill you in on what's been going on.
In no particular order, here's the scoop:
- We've been beefing up our crawlers
- Balancing freshness against growth, and
- Wrangling ornery hardware
It is difficult (at least for me) to comprehend 100 billion of anything. We've been in the "big data" business for a little while now, but to deliver an index of this scale at the quality we believe you deserve, we had to revisit some of our basic assumptions. One is the total rate of pages we crawl everyday. Not only was additional hardware needed, we had to improve our strategy for "politeness": the method by which we rate-limit ourselves to be a good web citizen, showing courtesy and empathy to our neighbors. Over the last four months, we've accomplished that goal and are now able to crawl well over double (and soon triple) the amount of pages per month compared to last year.
Simultaneously, we needed to improve our pipeline for turning crawl data into a final index, something we simply call "processing". Before January, our indexes always contained data from only the prior month of crawling. Depending on the exact duration we took to process the data, it was usually somewhere between 4-7 weeks old by the time it was released. Before any crawler improvements had been realized, the amount of data that we could collect in one month was simply not enough to get us to 100 billion URLs. So, we widened the window to include all data that was crawled in the last 2-3 months. That's why in January, we were able to launch our largest index to date at 58 billion URLs and again in February at 66 billion URLs.
In the December to January time frame, we started to see an increasing number of hard drive failures during each processing run. It turns out that spinning platter hard drives are one of the most fragile parts of a computer. Therefore seeing one or two failures each run is just a natural fact of life. To account for this, we check point every step of processing, so after such a failure, we can simply restore from S3 and continue from the last completed step. These backups are typically 100-200 terabytes large cluster wide. Historically, the restore process took no more than a day, so even if it happened 3 or 4 times in a month, we would only rarely slip our release date.
In December, that story changed. We experienced five hard drive failures and an additional machine failure, which caused us to restore from S3 a total of six times. Thus, we ended up shipping the index a few days late. In January, we again experienced five hard drive failures. Worse, several of these failures were clustered in one of the longest steps of processing. Since the index itself was growing, the restore step ballooned into multiple days each, causing us again to slip the index date. Additionally, we determined that the raw data stored on each processing machine would no longer fit within the local storage (or ephemeral storage in AWS's parlance), if we continued to grow the index. We engaged Amazon, our infrastructure provider, to see if they could help us deal with these issues.
As we were already using the largest available amount of local storage in AWS, the only real option was to move to Elastic Block Storage (EBS). This presented two downsides: performance and cost. EBS is not truly local, thus it entails a performance hit. It also costs both for total capacity and the rate of usage (or "IOPS"). Compared to ephemeral storage which is already included in the cost of EC2, this made a huge difference in the total cost to produce our index. Coincidentally, back in January someone asked on Quora how much producing the Linkscape index cost. Moving to EBS would double the total; since, in addition to the EBS cost, we needed to move to more expensive hardware to utilize faster networking. Still, we had no other choice, so we pushed forward. Additionally, the move to the larger machines required us to upgrade the operating system and change the virtualization layer that we were using, incurring more development time, which we paid for at the start of the year.
The good news for March was that after starting the index in late February, we realized that we were very likely to exceed the 100 billion URL goal that we had initially set out to achieve. In fact, we were expecting to release a new index at the end of March that contained over 150 billion URLs and over 1.7 trillion links. With that goal in hand, we started planning out the design and style for Rand's new facial hair, as was mentioned back in January's blog post. Believe me, we had some good ones.
Unfortunately, even though we increased the computing power of the machines that we were using, the performance hit of moving to EBS and the huge increase in data size caused way more growth in processing time than we had initially expected. Additionally, even though moving to EBS was supposed to be a more reliable solution, we still experienced hard drive failures. Fortunately, we had also switched to a higher redundancy RAID configuration, allowing us to save time when these drives did fail.
Sadly that leaves us in the state we are today having slipped past the end of March release date with no index ready to launch. So now, Rand's facial hair is safe, and we've disappointed our customers. We're really sorry for any trouble this has caused, but we assure you that we're trying to get things working as quickly as we can. We have be working tirelessly on several alternative solutions for increasing the computing power of our processing cluster to get a new index out as quickly as possible as well as trying to avoid this situation in following months.
It's nice the Moz community is so forgiving, but this isn't a novelty tool for me or many other companies.
I have had no choice but to utilize ahrefs service several weeks ago. They have some nice unique features, but nothing with the integrity of PA/DA data paired with the historical movement. The upside, they have a product that isn't so damn stagnant.
I also feel bad for companies like Conductor who utliize linkscape as their exclusive provider. Think of all the vendors who rely on linkscape for a quality product. Apologies and explanations are nice, but if you pitch a product with a specific timeline and don't reach it.. does an explanation solve the problem or make the customer truly happy?
Linkscape is the only reason I pay for SEOmoz anymore and everytime my google calendar updates with a new date for linkscape update, my stomach turns.
Thanks for your honest feedback - I know how difficult it is when we have to move these updates. Unfortunately, this has happened a few times already this year and so we all felt the community deserved to understand why.
I know it doesn't help the immediate problem, but we are working nonstop to get these launches back to a dependable launch cycle.
Do you plan on getting it back on the 30-day interval as soon as the issue is resolved?
Definitely, that is our top priority right now. We are looking at several different options, but we know how important it is for all of you to be able to depend on the 30-day schedule.
You can't do better than being honest to your customers.
You said you wanted feedback. The only feedback I can offer is maybe stop giving us dates!!! I'm tired of going to the update calendar and having my heart sink. I no longer trust any date you say.
On a sort of unrelated note, I am honestly trying to understand what you are offering that is worth $99/mo. OSE is free; reading all your valuable blog posts is free. Am I just paying for that thing to crawl my site? Aren't there free tools that do that???
I think SEOmoz has the most reliable and best SEO information, and a weekday doesn't go by that I don't read your blog, but I'm starting to reaccess being a pro member. What exactly am I paying for? And that's not retorical if someone wants to enlighten me.
Totally empathize with your frustration. I'm in a similar position as a user of the data myself. SEOmoz PRO offers a bunch of stuff - Q+A, the Web App, the PRO Tools, Guides, Resources, PRO Perks, Mozbar, etc. - and we've been working hard to improve many of these (e.g. Q+A had recent upgrades, the web app has lots of upgrades including the new vertical rank tracking, the KW Difficulty/SERPs analysis tool is now best in breed, GA integration has KWD suggestions built in, etc).
Linkscape is obviously a big part of what we do here, and this index is slower in updating than we'd like, but the metrics and links there are still much better than what the search engines provide and a new index is coming as soon as possible (trust me, our engineers are working their butts off trying to get it out). Just a few more days should do the job.
Thanks for understanding and sticking with us, and certainly if you decide to move on, we empathize and hope to earn back your subscription when updates get faster (we're working toward a 2-week index cycle in the months to come).
Let me start by saying I am a huge fan of SEOmoz, the community and attitude towards SEO is fantastic on here.
However I feel hugely disappointed in the manor you have dealt with this situation. The whole brand and message SEOmoz portray is based around giving customers a great experience on the web. But with this you seem to be not only lacking that, but also general business sense.
While I accept this is far from a simple problem and I do truly believe you are doing everything possible to solve it, the bottom line remains the same. If a product is not reliable or ready it should not be charged for, it doesn’t matter what it is or who you are. I don’t understand in what world you believe it is acceptable to charge for a service and not deliver it, but worst of all not offer any form of compensation or gesture of good will. (in response to comments above it is not about what is economical to you, it is about what is fair to the customer)
I have been visiting SEOmoz on a daily basis for months and took out a free trial last month. Unfortunately I am reconsidering continuing the Pro membership not because of the problem itself, but because the manor it has been dealt with does not give me confidence for any future problems.
My advice for you would be to practice what you preach
Truly disappointing. Mentioned it to a client - their response? 'Not our problem - sort it out' Unforgiving or justified? hmm.
Is it not possible to do a 'most of the web' monthly and an 'all of the web' bi-monthly or something, so at least we have something to get our hands on every 4 weeks?
Not confident of latest date being met, I must admit :(
This is beyond disappointing now - if I provided half a service/product to my customer I wouldn't get away with receiving 100% of the fee. Waste of money.
Not only it is not updated on time once, we now have another 3-7days wait. Any more?
I'm not even an SEO company - I would fuming if I was and had clients with no data. Sorry if this sounds harsh, I think it will be echoed by many though. I don't have to appreciate hard work and input. I do appreciate receiving what I have paid for though.
The same goes for Rank Tracker too which I raised an issue about. A daily of 200 is only any good if it will actually work. Unable to retrieve your ranking pops up more than the actual ranking - evening/weekends are more reliable but Mon-Fri is awful at times. Again apologies if this comes across as harsh - I throughly recommend SEOMoz to anyone, and likewise whenever I speak to anyone about SEO they always ask if I use SEOMoz. It's a fantastic resource, but it's only any good if it works. Fort me the blogs/tutorials/webinars are a bolt on - the research tools are why I signed up and have used the site for the last year.
I get that with rank tracker as well, the only way to fix it is to refresh the page and try it again.
For me it seems if you update a few rankings it fails (more than 3) not at the same time, I am clicking then waiting for result then clicking again. If you click multiple it fails straight away.
Sometimes you need to refresh multiple times - depends on the time of day. I've never tried to do multiples - I think it would die lol! If you can get it to do 1, it will normally do about 4 before it dies again.
Very annoying. Find myself using Keyword Analysis if it's just a single k/word I want to quickly check.
Seriously, the delays are getting really annoying now.
I know you're all working hard and trying to make this update powerful. Just keep in mind that some of us (me) use data from the Linkscape Index in quarterly reviews with clients.
Hey Jonathon,
We understand and take the impact to our API community very seriously. There are a lot of changes that are going on around here to resovle this going forward. The engineers have come up with some creative ways to finish up the current index and we are tirelessly working on getting these launches back to a reliable schedule. We definitely know how much you all depend on that.
Thanks,Carin
Carin,
Thank you. I'm still committed to the SEOmoz product and probably always will be. SEOmoz had me at hello many many years ago. Happy Holidays.
-Jonathan
hmmm...so to be clear, we should get an update over the next week but when we do it'll be 9 weeks out of date?
It's now May, a 1/3 of the way through the year and there's been ONE update? Please have a rethink about how you're doing this stuff, seriously considering carrying on Pro membership now alas.
Yours disappointedly.
Indeed, I work with some websites who were involved with blog networks that have since been nuked and most of the sites taken offline yet OSE still reports them. That's fine - I can understand that, but as it's happened within the last 8 weeks should I expect them still to be included when the update comes through?
Also to add - if the next update is delivered on time (1 month after this one is live) would we not have waited around 13 weeks to see 'up to date' information (which even then is up to 4 weeks old).
I may well be completely wrong and just confusing the matter - please let me know if so.
When it comes to customer satisfaction, I must agree with everyone here, two months is too long. Especially since SEOmoz customers and everyone who is using the API is paying for it monthly. You guys definitely need to make sure this process takes less than 30 days.
However, talking from experience, dealing with large scale computing projects is not trivial. One of the most important things is to start with a good design, one that is scalable and allows further growth. If you don't have that, then you need to change it. If this is what happened at SEOmoz in the last few months, then I believe it is worth waiting for it.
And I don't believe asking other people for advice or adding more people to the team would be beneficial right now. It will only slow things down.
Oh, and one more thing. I personally know many of the people involved in creating Linkscape and I believe they are truly capable of making this happen.
Keep up the good work guys!
Yes yes, good to be honest and such. But all I heard was, "blah blah blah, science science science, BIGGER".
Good luck :)
Haha, well Starlen I loved the portion of your comment, you are awesome with your words...
"blah blah blah, science science science, BIGGER"! :)
More than disappointing.
Clearly, you bit off more than you could chew. That was the real mistake. For all its candour, the post is NOT re-assuring. It remains unclear that you have truly learned from your mistakes. It looks like you may be persisting in your folly.
As I wrote when a previous update was late, many of us are going to disappoint clients.
What we need above all are regular updates. You have now established a pattern of failing to deliver. Not good.
Put another way: first, ensure your can deliver the basic level of service. Then, and only then, worry about upgrades.
One of the things that make SEOmoz great is your attitude to continually push to be better, which is what the ambitious index target was about. In the long term, this will be better for all of us who use your tools and data.
Its inevitable that any business operating at this scale will suffer some hiccups, the measure of awesomeness (MoA) is around how you deal with it. Here, you made the decision to push forward with your amibitious index at a monthly cost increase of $200-300k, were decisive and speedy about the move, and perhaps most importantly, you were transparent aboutt it. :)
Thanks for the update -- I'm looking forward to this index. Mmm.... data. :)
Please don't think you've disappointed us. We can see how hard you work and you're effectively doing the job of a search engine with the index. Thank you for keeping us all informed and for putting effort into a quality product.
We look forward to the update, but only when it's ready. Thanks again :-)
And Happy Easter!
We can appreciate how hard they're trying and still be disappointed.
I’ve actually been disappointed with the linkscape index for a long time now which has resulted in me using majestic SEO’s fresh index.
With your index being so slow, I can’t check out any new links that my competitors have built, and if anyone has removed links that point to my domains. This is especially important to me as someone that constantly buys websites.
2 months is not acceptable in my opinion especially when you guys charge $99 per month.
Ever since yahoo site explorer stopped working the only real replacement was majestic SEO, but why can’t seomoz have a fresh index as well? Does it take 2 months to calculate all of the metrics? I really don’t get it.
I’m sorry if I’m sounding a bit harsh, but I’m sure you guys appreciate honesty.
I actually totally agree. While our metrics are often better (PA, DA, etc), our index size, freshness and link data isn't as comprehensive. MJ has their own issues with a lot of "junk" in the index (something they certainly admit to and are working on), but right now, Linkscape is not "acceptable." Believe me when I say that many of the smartest people I've ever met are working ingloriously long hours to fix this. The scale of the problem is massive, but once we get there, I believe we can rebound and be 2-3X better than we've ever been on all fronts - freshness, size, quality and metrics.
For now, though, I'm using a combination of Majestic and Moz myself and I wouldn't suggest anyone else do otherwise.
Uneasy lies the head under(or that wears) crown...
The wise will understand :P
The solution in my opinion is to use Linkdex, which re-crawls all of the URLs that Majestic informs them about to check they are still there.
This way you get a much bigger index size than SEOMoz and remove the issue of Majestic being full of junk.
I still use Seomoz as well, but I think they still have a way to go on the link data side.
I appreciate your honesty Rand. For smaller businesses like our own paying out for two sets of software is a huge commitment - and something our own cannot afford to do at present. I do appreciate your hard work and I don't want to have to consider our options going forward.
I think if I was using the free version it wouldn't worry me as much, because I believe we shouldn't complain about free things, but the fact that we've paid means we really need it to be working.
I look forward to some good news soon :) Thanks for listening.
Well sometimes crap happens its not new.. but hey u get to see who your real friends are in times like that...
You have given a lot SEO community..If not to them you sure have given to me.. I will stick to you guys no worries..Just don't delay more please ok?
Btw Just saying
you guys should give some discounts to ur friends who stick by your side :D or some do follow links hehe..
Thanks for providing a lengthy explanation about the delay. The important thing is that Rand's beard is going to be alright. Good Friday to ya'll Mozzers.
Whilst it's admirable that you guys have given such detailed response, allowing us to inderstand your frustrations, this is a service many people are paying for.
Will the SEOMoz team be refunding or adding extra months credit to those who have been pro members during this time?
As many people have pointed above out two, and perhaps now three month old data, with nothing to show clients is more than a little frustrating. It's just not good enough!
Luckily for me I have understanding clients and paid for other tools that i can afford to leverage. I fear many others may not be in that same position...
Hey Charlie,
While we understand your frustrations and would love to make up for this delay, it simply isn't economically possible for us to credit all of our users who have been PRO members during this time. We also have a number of other tools and services that are included in each membership so I'm afraid that we won't be able to offer a credit for this delay.
Again, our deepest apologies. Our team is working extremely hard to get the index updated as soon as possible and we believe it will be much more consistent in the future. Thank you for bearing with us during our growing pains.
It makes more economical sense than a wave of customers cancelling their subscribtions.
IF - this happens again after this update and no goodwill gesture is made I would have wished, if I owned/ran SEOMoz, that I had the hindsight to be proactive rather than be in a situation where I would need to be reactive to ensure my customer base doesn't jump ship.
As mentioned in my above comment too.. why is Rank Trackers reliability so poor?
Hi Allan - rank tracker's been largely deprecated (as noted in the messaging on the tool for the last few months). The rank tracking functionality in the web app, however, should be very accurate (we've been told and found that it compares quite favorably to anything else on the market) and has vertical results included now, too.
In terms of cancellations - we definitely understand that this is a key part of the service for many folks. OSE and the Mozscape index are still functional, still better than most data you can get from engines, and will be fresher, better, bigger and more accurate in the near future. We recognize we're still a startup, still undergoing growing pains, but hopefully doing good work that will pay off in the near future.
Hi Rand,
The accuracy is great, it's the freezing/not responding which is the problem. Deprecated is not what the message says though, the implication is it will be updated not retired? K/word tracking within a campaign is usually a week out (and we have a 300 monthly limit), so I find Rank Tracker very handy for checking new content which is a few days old but I don't need constant rankings for it. I noticed the processing icon disappeared a few days after I filled in the survey and reliability (maybe a coincidence) slightly improved, but it's still has performance issues.
With regards to the cancellations I appreciate there may not be a better alternative but that doesn't imply people are expected to settle for less than they originally paid for. I must stress, the lack of fresh data in OSE doesn't bother me enough personally to cancel the subscribtion, as I'm not an SEO expert and don't make use of the data provided in a structured or knowledgable way - but many others will and I thought it was worth raising the point.
I've got to side with Rand and SEOMoz on this one. OSE is usually updated once a month... they missed a month. Does it cause issues? Yes. Does it require a little bit of explanation to our clients? Yes.
However, the strategies involved in increasing the metrics that the Linkscape Index gives us visual and viable confirmation of have not changed. Keep doing what works, and next week it'll be like an old Spearamint commercial when you see the updated data for the first time in two months...
Double the pleasure, double the fun.
Also, for the guys talking about refunds... make no mistake that the $99 you spend here is absolutely the best ROI available in the industry, OSE or not. This is my first official month ;-) as an SEOMoz Pro Member, and the next month should be free. Not because I requested a refund, but because I spent a little time contributing to the community and answering questions. 200 Moz points = one free month of PRO. The free month withstanding, I find that spending a couple of hours a week helping people will inherintely lead to business leads, and educating others is one of the most effective ways of educating yourself. These guys provide great tools at a great price... Yeah you didn't get what you paid for in March, but you'll get three times that for the rest of your membership... as long as Rand doesn't jack up the price. Then we've got beef ;-)
Thanks,
Anthony
It isn't economically possible for me to pay for something and not get it for a few months either - yet that is what I have been doing.
Sorry to be cranky about this. I do appreciate the effort that goes into the wide variety of services and info provided by SEOMoz. But I do I feel a little cheated and somewhat embarrassed. I work with a lot of small biz clients who previously hired shady SEOs, spent a lot of money and got absolutely nothing. I have often pointed these clients in the direction of SEOMoz as a shining example of the good side of SEO, and recommended the site and its tools as a means to do their own research, learn more about SEO and even do some auditing of my work (or whoever is doing their SEO).
I don't understand the refusal to do at least some kind of good-faith gesture like a short extension of subscriptions or something like that. Your pro community has paid for X amount of services, and received less than X amount of services.
"We also have a number of other tools and services that are included in each membership so I'm afraid that we won't be able to offer a credit for this delay."
This would imply that those other tools are worth $99/mo and your index should be free!
Thanks for the reply. It's just that it seems this data is what most people are paying for. Whilst there other other services you provide, this is the predominant reason I use Moz.
I just wonder whether you guys and Rand perhaps bit off way more than you could chew with this massive update, where most of us would have prefered smaller more regular updates.
I would have thought after issues that are completely out of our hands (and perhaps yours) that you might consider some recompense in terms of an extra months credit as goodwill for us sticking with you through the inconveniance and growing pains rather than risk lose us as long term users.
Anyway, glad you guys are working at it, and sorry to be here complaining.
While I do love SEOMoz, 2 months of not getting a big chunk of what I paid for is troubling.
Well, after such a lengthy explanation and admittance of everything, I dont think its disappointing at all. Wherever we have to rely on machines, I think we need to keep a very good margin. There is an Pashto proverb that if you want to keep camels in home, you will have to make large size doors. So if we are dependent on machines we have to keep our hearts open.
One one hand though, I think there is a matter of greed involve in it, you could have continued at the slow pace without having problems and may be you could have done much better and reached 100billion gradually, I dont think its a very good idea to do such sorts of experimentation when you know your customers are waiting for the update.
The explanatory post here in SEOMoz is worth reading and much informative, but since you have gone into technical depth about the sort of storage you use for processing, and how you crawl and process everything, I think it would be better if you also included a small portion in the post about how other crawlers do their crawling. You could have given us a slight Idea about how other companies perform this task? May be a slight overview of what technology does Google and Bing use could be much beneficial for us in know what's going on behind the screen and whats the difference between both sorts of crawling.
And since you are already on an extensive overhaul and things have delayed, can't you add a fresh index into opensite explorer as well? Like the one that majesticSEO!(I have limited knowledge about how they manage and crawl for that fresh index, would appreciate if anyone can explain a bit).
Anyways, it has been delayed, and people are disappointed, but now we expect 150 Billion in the next update... One link less than 150billion, and Rand, you are doomed... :)
Wishing the SEOMoz Team a best of luck, Work harder (and smarter as well)!
Hey Asad,
That is a great proverb :) You're right, it would have made a lot more sense to have gone slow and let our cralwers slowly grow to 100 billion organically, however, late last year we discovered a design flaw in our crawler system. The engineers realized our crawls were actually getting iteratively smaller. This was one of several factors that pushed us to make our crawlers more ambitious.
Open Site Explorer is pulling data from this same Linkscape index, but we have tossed around a few ideas of producing a smaller, fresher index. I think it would be incredibly useful for a lot of our users. I'm not sure if we would be able to squeeze it into the road map in the next several months, but given the problems we've run into as of late, it's definitley becoming a higher priority.
Thanks for the great blog post suggestions! I think those would be make great posts and I will forward these suggestions on to the team!
Thanks,Carin
I can feel how disappointing it would be for those with the subscription, at the moment, I dont have it, but I know its really frustrating when you pay for something and you dont get it on time. I just hope SEOMoz engineers sort it out soon and they come back with a massive 150 Billion figure this time.
Fingers crossed...!
Very disappointed. I have been chasing this for months and each time we get fobbbed off with server/data manipulation excuses. We pay very good money for this service and expect it to be working when promised.
It would also be appreciated if you contacted you customers to personnally explain what is happening and what financial responsibility you will burden for your failings....after all that is what our customers expect of us!
When I look at the update calendar, why is the update spanning six days? Is it because you don't know which of those days it will be?
Yes why is update spanning 6 days?
Hi! As I mentioned in the update, it will take several days to complete the index. During the scheduled time we'll launch the links and url metrics first, with the anchor text shortly after. As soon as we have more info, we'll have another post and will Tweet updates. Thanks!
See - you can still learn things at Moz. Today's lesson: "How to Increase Your Twitter Following by Underdelivering"
1. As much as I don't want to encourage it, Moz's statistics are worth it even if they are several months out of date (hell, how often is PR updated?)
2. You increased the API rate limits, so I am ecstatic
3. Is it "Linkscape" or "Mozscape"?
Glad to hear you're excited about the rate limit increase - we were really excited to finally be able to increase them!
It is now Mozscape - you'll see even more improvments this year, but we started with a fresh name and some major performance enhnacements allowing us to decrease throttling for our paid users.
This just in:
Google toolbar PageRank is now more fresh than linkscape.. wah-wah-wah!
Seriously though, on the next delay can you tell my clients/boss that their metrics are not the same, the problem is on your end? That would be swell.
Signed up on a free trial looking to see how quickly you guys would find new links I have prevously build.Looks like my trial never coinsided with an update:( Will reconsider when rolling updates are live.
I'm so sorry that an update to our linkscape index didn't happen during your free trial! I'll be following up with you through our helpdesk to see what we can for you.
I'm a huge fan of SEOMOZ, it's blog, Q & A, videos and community. Your staff communication is amazing, but I can't work without fresh OSE data.
I don't believe Majestic is even close to as good in terms of metric etc.. but I can't wait much longer, many of my competitors have change ranks in the 2 months period with no fresh data. I'm currently blind. I really don't want to pay for two different tools.
I would recommend you look at other solutions for solving this problem other than pure computing power and trying to crawl the entire web, why not crawl your pro clients websites & their selected competitors website + their search queries as a priority. You could even use Webmaster tools links for updates or team up with Majestic for the crawling side and you could concentrate on the link metrics?
Anyway I'm talking about stuff I have no idea about, just though I would make a suggestion.
I'll be so happy when I see the next update! Really looking forward to seeking how competitors and my DA is going.
It is frustrating, especially as I've only been a member for a few months and trying to dig my teeth into it. There's a lot of great content on SEOmoz that I've used but I'd say a good 50% of the benefits of the pro account for me (and for others a lot more I'm sure) are from the Linkscape updates so perhaps a partial refund to pro members for the previous month would be fair.
I can certainly empathise with the technical issues but for people like myself, this subscription is quite expensive and it becomes quite hard to justify unless there's regular updates.
Hope you manage to get it sorted for the 27th.
Can't wait for the next update - go team!
Thanks for the update Jen.
Looking forward to the bigger index.
Thanks for the update. I acutally had an amusing converstaion with someone because of this delay.
A college of mine ran an opensite explorer check my client's site. They didnt' belive that I had some of the guest blog links that I was claiming to have...
Obviouslly, a simple Google search query with a links: opperator was enough to settle the dispute.
This brings up a good point. Many non-subscribers have no idea that OSE is not up-to-the-minute. I have some clients who I told to check OSE and a few other places to check for new links, research competitors on their own if they choose to do that, etc.
Maybe a prominent freshness date on OSE would help that. Otherwise, you'll have casual users looking at OSE, then checking elsewhere and seeing that things look "better" on the other link checkers that show more, and more recent links. The average site owner probably doesn't understand care about trust, authority, mozrank and all the other cool metrics.
Hey Nick,
This is a great point! I'll add this to the list of enhancements for OSE - I'm sure it wouldn't be too difficult to find a place to display this.
Thanks!Carin
Not getting important data on time is quite frustrating, especially if you try important stuff like anchor text distribution, linking root domains and the total number of links you created to a particular website.
I never did managed to get all the data from OSE (compared to majestic I usually get 25% of all links indexed). However, I have to admit I am still using it, just because of the Page Authority and the other really cool metrics.
I really dream for the time when I will be able to see more links in the OSE...
Thanks for the update. 150 billion URLs and over 1.7 trillion links sounds impressive. I hope there will be no further delays.
I hope SEOMoz dont break anymore dreams... :)
Whoa-oh! I just decided to try the FREE version of Ahrefs since I've heard about it a lot on here. I'm less upset about the OSE delay now for two reasons:
1) Ahrefs found backlinks that we've had for a long time that OSE NEVER detected. That can hold me over for now.
2) Because of #1, I understand now why you took this giant leap to try to make your index better--it was leaving so many lniks out.
I'm no longer upset with you for the delay, but if you don't deliver by tomorrow I probably will be, because that would just border on ludicrious.
Totally understandable. Crawling at that level is nothing easy. Google had alot of issues early on with the crawling, and I bet they still do.
This is a hard problem, to be sure. The description of what's going on makes it sound like your engineers designing the system may not have built many large distributed systems. In particular, the description of the very batchy sounding all-or-nothing start over on a hard disk failure is scary for a system of this size.
These kinds of systems need to be seriously decentralized and they need to expect that failure will be frequent and commonplace. Read, for example, the accounts of how Netflix went to Amazon.
I recommend getting in touch with some folks who've done some seriously large scale Cloud systems who can lend some advice. They're out there and compared to the expense of being late or disappointing customers are pretty cheap. It's too late to use their advice on this go-round, but you're going to continue to see pain in future go-rounds if you don't get some help.
Cheers,
BW
You are exactly right - this system is much too large for it's current architecture. We have some of our best engineers on the Linkscape team with a lot of experience in distribuited systems, but, unfortunately, they have inherited a code base that has outgrown it's architecture.
Our long term plan is two fold:
1. Resolve our operational issues so we can reliably get an index launched to our community on a reliable 30-day schedule. There are several options we are investigating.
2. Redesign this system so it performs at it's current scale. As our index size has grown, the system has not been able to scale appropriately. The path we are going down for the new system would be decentralized and more of an incremental system. This is our long term plan.
I realize that doesn't help much with our current position, but I hope that gives you some faith in our long term plan!
Thanks,Carin
Disapointing. Thanks for the update though.
The only thing that holds me from buying pro account is that linkspace is slow with growing links db
#OSE is not working?? it never does... and regardless of the problems I have had too many times when I can't actually get a report even if it is using the old non-updated data... would still be nice to get it...so I can WORK with something...
Thanks for keeping us up to date Jen.
Sorry the schedule got knocked sideways again & spoiled everybody's weekend :(
Big thankyou to everyone for all the extra effort to get it sorted.
Man that sucker is BIG! :)
Sha
I'm definitely addicted on SEOMoz tools )
I hope the Linkscape will be updated once a month.
People/businesses make mistakes all the time. Even when dealing with a large number of customers. It's how we/they deal with those mistakes that makes the difference. The honesty and transparency is much appreciated. Now get back to work ;-) I kid! We look forward to the updated index when it's ready.
What an enormous task to undertake. I hope there are not any more unforseen glitches. Makes me want to pull my beard out. :-)> Happy Easter!
Now Look... (I use that opening to celebrate the current election session in the U.S.) We can all use the time we would have spent dissecting the new link scape data to write and produce some really great content!
Our little company uses Google web master, traffic to the site, and conversion data with customers. All of which is at your finger tips in a nano second. Admittedly our SEO clients are in general not that demanding. The clients that are that interested in PA/DA and all of that, we set up with their own SEOmoz account and train them how to use it. That way, the responsibility for the updates are seen as Rands responsibility ,and not ours. I mean... Rand is collecting all the dough right? ;)
I find myself spending far to much time looking at links, and not enough time creating traffic for customers anyway. Maybe we should see this as a blessing in disguise.
Rand, maybe a free coupon to all your paying customers for something cool. Like, a free go cart racing course, or a free sky diving lesson, or a drawing for a date with Roger. Yes, that's it. Lets have suggestions on what free coupon SEOmoz pro's can receive for compensation.
P.S. ( I use to have a pro account, but then i lost my job at Denny's and had to cut back)
Update to the index deployement: The full index is now live! This index is exactly the same data as what was released on 5/1, but includes the updated Anchor Text views.
Thanks for the explanation. Happy Holidays :)
Still I'm awiting to update.. By the way thanks for explained
Well guys. I still love you.
I've only been a member for about 4 months and things haven't been working properly for over half that time. In a previous post it was stated that providing compensation to current paying members was not economically feasible. Being as 18 million in capital was just acquired today according to your blog. Does that mean we can now expect some sort of compensation for the last two months we paid a membership fee for a broken service?
Free month for all paying members?
Given that the same post reveals the company had to spend almost a half a million dollars in the month of April alone to keep work on the index running...maybe not?
haha, I can feel the pressure mounting.
Hi everyone! I've just updated the status at the top of this post. Whee! I'm excited about this one. Here's the update:
Hooray! The new index went live today and it's BIG. Check out the latest blog post for all the information. We all thank you for your patience and support (and understand your frustrations).
HD failiures on a crawl should auto swap out to a hot spare autoatically?
One question - how fresh are the links in the index likely to be? (i.e. Have you completed the crawl for this run and are now just calculating metrics, or is everything still ticking along?)
Hey Matt,
Great question - when we're on our normal 4 week cycle, the freshest data in the index is going to be about 4 weeks old. Because this index has taken twice as long to produce, the freshest data will be 8 weeks old, unfortunately. We do have another index currently processing that we will launch as soon as it is complete.
Okay, cool... good luck getting everything fixed! :)
Thanks for sharing but you can make your inforrmation more attractive for clients......
Thank you for keeping us updated. I don't think most people appreciate the complexities of what you're trying achieve and they certainly couldn't do it themselves. It's important you make sure you get everything right and you're able to support the system, make it reliable and accurate. I'm really looking forward to the new index and what it reveals.
So how old will the data be when it goes live? (assuming it indeed goes live between 4/30-5/5?) 9 weeks? I mean if we're waiting another week for fresher data than that, that's cool. ;)
Unfortunately, the freshest data will be 9 weeks old.
We do have another index processing just behind this one! We kicked it off the middle of April and we hope to be able to launch a few weeks after we get this one out.
If the freshest data is 9 weeks old how old is the 'oldest' data.....come on this isn't good enough...meant to be monthly, then we are told delayed, then we are told that even when the data is run it will be 'out of date'....the point being?
Maybe there should be some kind of discounts, maybe be we should all be mad because they didn't deliver, maybe this, maybe that. I partially agree......however, I strongly believe, like Gulit, that we should be thankful that we have not only a product, but a company that thinks so highly of us to render an apology and an explanation.
Besides, are we not partially to blame if we are SO dependent on one tool that we have to actually go to our clients with our tails between our legs?
Mozzers, you rock. Keep up the good work.
Wow, I never realized the linkscape updates cost that much money! What you guys are doing is truly impressive, keep up the good work!
To be honest I have held off on leaving my comment on this post because I felt I needed to put some distance between myself and my first read of some of the comments above.
Frankly, I am saddened. Not by the extra wait for publication of the Index, but by the constant reinforcement of a loss of humanity in our world as the almighty dollar becomes more like a god than an asset.
I was lucky enough to be born into a world that sought to acknowledge hard work and give thanks for what we have. A world that cared about the things that others gave up while they worked to achieve something important. A world that revolved on fairness, honesty, courtesy and respect for others. I can only thank God I am not handing this self-centered, discourteous and ungrateful world on to children of my own. For the children of others, I can only be saddened and ashamed to have been a part of this ugly decline.
To Brandon and Carin:
Thank you for for showing me the courtesy of providing an honest, detailed and educational(!) explanation of the reasons for the delay in updating the Index.
To Rand and everyone else at SEOmoz:
Thank you for the Index; the tools; the testing and information sharing; the Q&A answers; the technical support; the thought leadership; the opportunities for participation and exposure; the smiles; and the unexpected surprises. In addition to all of that, thank you for your efforts to improve the Index and to resolve the problems so that things will be better in the future.
Most of all, thanks to you and your families and friends for all of the hours of free time and family get-togethers that you have forgone and will continue to miss out on, in order to find a solution that will benefit me.
Thinking of you all and hoping easier times come soon,
Sha
Everyone moaning, I understand your frustrations. I'm frustrated too. However, the next update seems to be great from what I can see. A big index which will give you sll more insight into your own sites and your competitors.
Bring it on!
Not long to wait now. Lets hope I have got an increase in links.