Hey gang - it's that magical time again when Linkscape's web index has updated with brand new data (for the second time this month). Open Site Explorer, the Mozbar and the PRO Web App all have new links and scores to check out. This index also features the updated Page Authority and Domain Authority models covered by Matt last week on the blog.
Here's the current index's metrics:
- 38,295,116,929 (38 billion) URLs
- 466,742,600 (466 million) Subdomains
- 125,007,049 (125 million) Root Domains
- 387,379,700,299 (387 billion) Links
-
Followed vs. Nofollowed
- 2.03% of all links found were nofollowed
- 55.57% of nofollowed links are internal, 44.43% are external
- Rel Canonical - 10.34% of all pages now employ a rel=canonical tag
-
The average page has 70.61 links on it (down 6.67 from last index; we're likely biasing to a different set of webpages with the broader vs. deeper focus of this release)
- 59.02 internal links on average
- 11.59 external links on average
As you can see, we're crawling a LOT more root domains - we expect to have data for an extremely high percentage of all the domains that you might find active on the web. However, because of this broader crawl, we're not reaching as deeply into some large domains (some of that is us weeding out crap, including many more millions of binary files, error-producing webpages and other web "junk"). You can see below a chart of the root domains we've crawled in the last 6 months vs. the total URLs in each index.
We work toward a few key metrics to judge our progress on the index:
- Correlations with Google rankings (not only of PA/DA, but of link counts, linking root domains, mozRank, etc)
- Percent of successful API requests (meaning a request for link data on a URL from any source that we had link data for)
- Raw size and freshness (total # of root domains and URLs in the index, though, as Danny Sullivan has pointed out, this may not be a great metric on which to judge a web corpus)
We've gotten better with most of these recently - PA/DA have better correlations, more of your requests (via Open Site Explorer, the Mozbar or any third-party application) now have link data, and we're slowly improving freshness (this index was actually completed last week, but didn't launch due to the Thanksgiving holiday). However, we are not improving as much on raw index size (root domains, yes, which we've seen correlate with other metrics, but raw URL count, no). This will continue to be a focus for us in the months to come, and we're still targeting 100 billion+ URLs as a goal (though we're not willing to sacrifice quality, accuracy or freshness to get there).
As always, if you've got feedback on the new scores, on the link data or anything related to the index, please do let us know. We love to hear from you!
I have been working pretty hard on getting new backlinks for one of my client's site and we have recieved around 10 new links from 10 different domains over the past two months and i have yet to see these effect my DA, PA, or even show up in OSE or my campaign link metrics?
I know for a fact that they are followed links as well. Am i missing something here? My client keeps asking why the work isn't effecting the sites authority and i'm not sure of the answer. Will these links eventually show up?
Thanks! - Kyle
"My client keeps asking why the work isn't effecting the sites authority and i'm not sure of the answer. Will these links eventually show up?"
No need to worry about PA/DA updates as long as you see improvements in rankings, traffic and conversions. For real time (almost) back link analysis i highly recommend the free tool linkstant.
I agree ultimately the best judge is the ranking/traffic and eventually the conversions. However, when your stress to them that you need to increase your DA or PA to rank higher and they never move, its kind of an interesting scenerio.
Managing client expectations is always difficult. If I were you moving forward, I would not stress that DA/PA correlates with rankings. Rather, tell them that it does and then let the results speak. It seems to me that DA/PA are not what they really care about, but rather more traffic, more qualified traffic, and more business/leads/whatever.
The hardest thing about consulting is always client/relationship management. I think that if you do this well, you ultimately win because you will have a better retention rate and happier clients (and you'll be happier too, because you won't have pissed off clients). So I'd try to redirect what they are basing "success" off of.
That being said, I agree that it can be frustrating when the work doesn't seem to be showing up. Do you also use Majestic's Fresh Index? I use it in conjunction with OSE (Majestic for link discovery, OSE for link analysis).
I do agree with the fact that client management is the most difficult part of the whole SEO game and meeting clients expectations and make them happy is a critical job especially when you are in the industry where not everything is in your control.
Thanks to Himanshu for the tool, i didn’t knew about that ;)
Great suggestion with Majestic, i hadn't used that before. Very handy tool even for the free version!
Yes you are absolutely right Customer Satisfaction is the best Success. But some time i am really confused about recent Google's behaviour.
Linkstant shows links only when someone clicks on that backlink. It doesn't discover any backlinks real-time. For real time referrer links, this job will be done by google analytic's new real-time analytics. I also recommend woopra for great analytics experience. I use both, both have pro and cons. Thumb down for your not useful comment. I lost my 15 minutes working with Linkstant.
I agree that the amount of traffic and getting the traffic to convert is the best judge!
I agree with himanshu. The PA/DA is a tool for reference. Rankings, traffic and most importantly conversions is how you should be measuring your success. As for your links not showing up in OSE, I've seen it vary depending on the site and site that you are linking from. Also, although SEOmoz does a brilliant job at trying not ALL links show up in OSE. Are the links showing up on Webmaster tools?
I suggest you keep up your great linkbuilding efforts and stop worrying about PA/DA. As soon as you stop looking they will show up. ;)
I understand and agree the point you both are making. I just have found it odd that i have yet to be able to influence these scores at all. The links i have grown show up in YSE and WT and i have even gotten a steady flow of traffic from each site as well.
I just found it odd :) Thank you though for the encourgement though!
As my good friend the crocodile hunter would say. "wait for it.... wait for it...."
Do just not report data from OSE look at numerous sites and most important look at link metrics from Google Webmaster Tools.
YES!
Client site that was completely killed a few months ago now visible in OSE :)
So good to finally see some numbers instead of the dreaded 1, 1, no data message.
From my point of view the update seems pretty sound Rand.
A few of our client sites have seemed just a little "off" up until now. The information in OSE looks much better aligned with what we know of them in this update, so generally pretty happy with the change.
Thank you to you and to the Engineering team for constantly aiming to improve the index, which becomes more valuable with every new release.
Sha
Any chance of seeing a more regular index anytime soon? Now that yahoo site explorer has gone I feel that magestic SEO is it's only replacement as they provide a fresh index.
Don't get me wrong I love OSE but alot can happen in the time an update happens.
Thank you, but i really confused about recent Google's behaviour, if you get me some reports about it,.....
Last days search engines give us more and more surprises. Not very good surprises.
Woot!! OpenSite is an awesome tool!!
When should I expect next Index update?
I am waiting to see my site's stats.
Hey! You can always check out the Linkscape Calendar to see when the next scheduled update will be launched:
https://seomoz.zendesk.com/entries/345964-linkscape-update-schedule
The Christmas holiday caused our next update to be postponed a bit, but it is scheduled for January 4th.
Thank your for your help :)
Opensite exproler awesomest tool..thanks for great tool cheers!!
Thank you Rand! It will be interesting to see the changes in PA/DA. Also, we are looking forward to seeing the impact of freshness and how the new PA/DA correlates with Google rankings.
I'm not convinced that this update is entirely for the better.
Whilst domain diversity is obviously preferable to raw URL count, it seems to have resulted in an exponential boost to the DA of any site which uses 'widgets', has a social component, or naturally accrues links from a high volume of domains (e.g., domain vendors[!]*).
Whilst these metrics obviously correlate very strongly with authority/rankings, it feels from my experience with the new data that the impact of this correlation is over-stated. Essentially, we're saying that even the most 'rubbish' social media site could feasibly have a DA value of 60/70/80/90 due to the kinds of (domain-diverse) links they organically accrue, while an authoritative niche site with a lower volume of more authoritative links might see a lower score?
/ramble
*I wonder if there needs to be specific consideration given to domain vendors, given the unique position they're in to affect these metrics in particular. Consider that many of the big vendors link back to their corporate sites from registered-but-inactive and largely value-less domains... Should these sites have as a high domain authority as they do? Food for thought!
EDIT: Obviously corellation is what it's all about here, and there's room for error - if all we can measure is propensity to rank, and it's all about domain diversity, then obviously there'll be edge cases and some sites will be mis-represented. It feels, however, (from an albeit brief exposure of only a day) that the scoring on this component seems a little too aggressive based on my observations.
"we're still targeting 100 billion+ URLs as a goal"
Awesome! Great job SEOmoz keep up the great work.
Yay! We almost reached mT 6 :D
Still need to work on that, but I believe that is good news.
Thanks for the updates. Gotta love when a great product gets better!