This month we're bringing you a special holiday treat: two Mozscape indices in the month of November! We just released the latest index, and you can now find fresh Mozscape data in Open Site Explorer, the Mozbar, PRO campaigns, and the Mozscape API.
This index is similar in size to the previous Mozscape index with about 76 billion URLs. The heavy computing AWS machines we moved to in October, detailed in Anthony's blog post, has saved significant amounts of time in our processing schedule thanks to almost no machine failures.
This time saved means more time for the Mozscape engineers to work on exciting projects, like tuning the final configurations in our own private cloud! We've been running a similar sized index in our private cloud located in Virginia alongside the index releasing today. It's running a bit slower as we continue to tune and dial the last pieces, but we hope to be running a hybrid processing solution early next year. Running an index in the cloud and an index in our own private cloud means fresher index data for you and our applications!
Here are the metrics for this latest index:
- 76,668,945,929 (76 billion) URLs
- 664,205,988 (664 million) Subdomains
- 136,202,352 (136 million) Root Domains
- 892,544,725,878 (892 billion) Links
-
Followed vs. Nofollowed
- 2.31% of all links found were nofollowed
- 56.61% of nofollowed links are internal
- 43.39% are external
- Rel Canonical - 13.91% of all pages now employ a rel=canonical tag
-
The average page has 73 links on it
- 62.28 internal links on average
- 10.54 external links on average
And the following correlations with Google's US search results:
- Page Authority - 0.35
- Domain Authority - 0.19
- MozRank - 0.24
- Linking Root Domains - 0.30
- Total Links - 0.25
- External Links - 0.29
This histogram shows the crawl date and freshness of results in this index:
As you can see from the histogram, this index has some pretty fresh data mostly coming from October and the first week of November. The freshest data in this index will be from 11/10 when we started processing, and a good percentage was crawled late October and early November.
As always, we'd love to hear your feedback in the comments - the Big Data team will be reading and responding! And remember, if you're ever curious about when Mozscape is updating, you can check the calendar here. We also maintain a list of previous index updates with metrics here.
Happy data pulling, Mozzers!
I'm excited that we launched this so quickly, and very psyched to hear that processing time has been reduced. Congrats big data team - will keep fingers crossed that we get to see this consistently! :-)
Boom! Well done guys. Keep em coming :-)
Awesome Moz Big Data team! Always love an update! Keep up the good work
You guys should buy Majestic SEO :)
their metric are way better than the Majestic trust flow
Love data. Thanks guys!
Its really huge! Thanks for updates :)
Great work guys. Thanks for update...
Great post carinoverturf, Thumbs up! :)
Awesome! Thanks Guys!
really impressive you guys keep getting better PS thanks for the new local service that is awesome
Many post on the internet on this topic, however in this article I have been able to better understand the subject.
Those are interesting stats - didn't realise so few sites used rel=canonical
Awesome!! Great work Carin.. Thanks :)
Thanks! I see some cool new recent links included that we didn't even know about :D
Awesome. More data to analyze :)
Great to see another update so quickly! Thanks guys.
Updates updates and updates :) making the work easy... thank you so much
Perfect...and keep going..:)
Forgive my ignorance - is there a way (other than manually doing so) to track a domain's stats over time?
The web app - https://pro.seomoz.org - does this by default for any campaigns and competitors you set up in the links section.
Nice work!
Perfect for thanksgiving! I love seconds.
That is a treat! Keep up the great work. :)
Thanks for the update! We are very excited!
This is great, thanks!
Loving this! It would be amazing to see it update this frequently on a regular basis.
Great work Carin, Thanks for update !
Updates are coming faster and faster!
Growing day by day. Best of luck and long live SEOMOZ.
Great!I got a question - I've had some great links that went live over 2 weeks ago.They are located on big websites that get scanned fast.
How come I still cannot see them via the OSE? How long does it take for a live link to get crawled by the mozcrawler?
Most of this crawl took place between October 1st and November 10th, so if you got links up 2 weeks ago, or even 4 weeks ago, it's very possible we crawled those pages prior to the links going live. You'd almost certainly see them in the next index, and if we can maintain our faster processing speeds, that would mean only 2-2.5 weeks from now (fingers crossed).
Do you think we'll ever get to the point where MozAuthority could be updated on a daily basis?That would be amazing.
Also, i have a very important question:We know that the mozAuthority considers how many backlinks the site has, but does it calculate the MozAuthority of EACH link?
Meaning, If my blog has 5 links from medium-authority sites, and your blog has 5 links from high-authority sites, will you have a much higher authority than me in general?
Thanks for updates. SEOMOZ Booming Day by Day!!! Hope next month many more URLs
OH no! It has pushed the December mozscape update from the 6th to the 20 something :(
It really just means that the index scheduled for Dec. 6th came out on Nov. 28th (a week early). As Carin noted, we're gonna keep trying to get indices out more quickly, so you might see another one in 2-3 weeks rather than the usual 30 days.
That would be lovely!