During the final day of SMX East, Nick and I sat in a fascinating session where search engineers from Google, Yahoo! and Microsoft all answered questions posed by the audience and filtered by moderator Danny Sullivan. I have, in my 5 years of search conferences, attended numerous sessions like this, but I must say that this was, if not the best one I've attended, very close to it. The questions were excellent and the responses were forthright, honest, direct and never condescending. It was not only refreshing to hear, but remarkably valuable.
Below, I'll share the top 6 takeaways I got from the session:
- Using Session IDs
Generally speaking, it's a bad idea to put session IDs in the URL. The engines mentioned that they have standard de-duping of session IDs in URLs (and we do it with our Linkscape index, so it's hard to imagine the engines could be less advanced), automatically canonicalizing such uses as SESSID, SESSIONID, etc. However, it was not made clear whether links that point to these URLs would be counted as part of the original page. For example, if I have the URL seomoz.org/blog?SESSID=123 and the URL seomoz.org/blog, those might be treated as a single URL for retrieval, but whether the latter would receive the PageRank/mozRank/link popularity score assigned to the former was less clear. To be safe, smart SEOs should, if possible, not use the IDs in the URL and, if they are employed, use 301 redirects to ensure proper flow of link equity. - Affiliate Links
Shockingly, when asked point blank if affiliate programs that employed juice-passing links (those not using nofollow) were against guidelines or if they would be discounted, the engineers all agreed with the position taken by Sean Suchter of Yahoo!. He said, in no uncertain terms, that if affiliate links came from valuable, relevant, trust-worthy sources - bloggers endorsing a product, affiliates of high quality, etc. - they would be counted in link algorithms. Aaron from Google and Nathan from Microsoft both agreed that good affiliate links would be counted by their engines and that it was not necessary to mark these with a nofollow or other method of blocking link value. - False Positives in Spam
Although the percentages are low, the engines generally agreed with Aaron (of Google)'s position that it does happen and they do request feedback if/when you think it's hit your site. With Google, you can send feedback through Webmaster Central, with Microsoft, Webmaster Tools and with Yahoo!, the Site Explorer Suggestions forum. - Links Are Still Huge
When asked if links are the primary signal for search engine rankings, the engineers generally agreed that, yes, it probably is. Aaron noted that links are a far less noisy signal than many others, including some forms of on-page keyword use and clicks in the SERPs. Sean from Yahoo! said that while it may not be the "most important signal" by itself, it's more important than, for example, title tags (which SEOs generally agree are critical to the SEO process). There was no mention that links would be fading away anytime soon - or that any competing signals had yet entered the marketplace as a potential usurper. - Sitemaps Shouldn't Be Ignored
At the beginning of the session, all three of the panelists talked about the value and importance of Sitemaps. Their advice included:- Sean from Yahoo! - Put really important pages in your sitemap, rather than every page on your site. Yahoo! considers sitemaps when figuring out which pages are valuable on a site, and if they believe this to be a trusted signal from a publisher, will use it more (the other engines seemed to agree).
- Nathan from Microsoft - URL structures in sitemaps are very important. Use the shortest, most authoritative, canonical version of the URL you want in the search engines' index in your sitemap file and they'll use that to help automatically filter duplicates and figure out which version to display.
- All three engines seemed to suggest, somewhat overtly, that sites often see an increase in search traffic when they use sitemaps. I'm not surprised - after building just a small portion of a search engine with Linkscape, we can certainly feel how and why engines would appreciate and bestow benefits upon those who properly incorporated sitemaps.
- PageRank Sculpting Should Be Tested
The engineers all felt that "PR sculpting" - the practice of using nofollow to flow link juice to and through a site to maximize and control how it was assigned to internal pages - was, generally, something that could potentially provide value, but certainly doesn't belong near the top of the SEO priority list. Instead, they all agreed with Danny Sullivan's position that it was something to test to see if your sites/projects receive value from it, because not everyone will.
Naturally, any time the engines give disclosure of this degree, more questions are bound to crop up. If you have any relating to these, or other issues, leave them in the comments below and I'll do my best to get them in front of the search engine reps at the next opportunity (possibly Pubcon, maybe before).
p.s. Hat tip to the very solid coverage of this session by Virginia Nussey.
p.p.s. One more item of value that I picked up from this session is in our PRO Tips section - A Half Dozen Brilliant Link Acquisition Queries.
Re: affiliate links - doesn't that solve the paid link conundrum? You can just call them affiliate links and pay out $xx for the first click......
Yep, I was really surprised about that answer!
It kind of assumes that affiliates only promote products and services which they 'believe' in.
Whilst this may well be the case in some instances, for the most part affiliates will promote heavily whichever product or service gives them the highest kick back...
"Affiliate Links" sounds even more legit than "sponsored by"...
Although Ill miss finding University Radiostations sponsored by www.buy{insert-random-hot-item}.com =P.
Isnt Affiliate (Link) marketing a way to promote your products/services and get a Link Back ?
As far as I understand it, the focus of affiliate linking (til now) has been seen as a way to promote the SALE of your product. (example is cutting a 10% kick back to the referral on a t-shirt sale).
I think the main interest is the ability to legitimize links (purely for Link Juice reasons) with something like "Affiliate, $5 per month with 1 lead minimum requirement".
Will sorta hit it on the nail.
Aaron from Google and Nathan from Microsoft both agreed that good affiliate links would be counted by their engines and that it was not necessary to mark these with a nofollow or other method of blocking link value.
And in one fell swoop this single 45 minute session has undone all the internet marketing suggestions, warnings and admonishments Matt Cutts has been promoting for the last 7 years.
Sorry Mr. Cutts.
There's more subtlety to the issue than you might think. The affiliate links have to be quality endorsements, not junk links in some footer or hidden div. See my comment below about "10,000" links vs a few hundred.
Why would some (Affiliate - i or you) put Affiliate links in some footer or hidden div. This wouldnt help neither of the both to earn $$$.
You are totally correct. I just want to make sure that everyone understands what Aaron, Nate, and Sean were saying when they suggested that affiliate links are juice passing and are OK. I don't think this goes against the spirit of the argument Matt Cutts makes.
Moreover, I think the engines reserve the right to evaluate these things on a case by case basis. So if you're trying to game the system, they can still knock the links or you right out of the index.
Given what little I know about such things, it would seem to me that if you removed the word "affiliate" from the above sentence and replaced it with words like "no followed" or "paid", it would still be just as true.
Links from valuable, relevant, trust-worthy sources will always be counted.
There might be slight algorithmic differences in how much link juice they pass, but they will always pass something.
I'm inclined to think you've got a good argument. The general case of links is that they are at least a measure of buzz which should count for something.
But as you start to look at some of the data, it's clear that there are shady links and good links. If taking into account shady links hurts end-user (searcher) experience, I don't think the engines are gonna count them. See my comment below about trying to rank with 10k shady links vs a few hundred good ones.
This also means that only trust-worthy (authoriative) figure, should be made a affiliate. every affiliate request must be manually checked as valuable and relevant.
Some months ago I wrote a YOUmoz blog regarding point 1: Removing sessionid from a URL, just in case anyone is interested.
And I am very much surprised by the answer given on point 2! This feels pretty much like a 180 degrees turn, doesn't it?
As someone who's heard the official line from the search engines and has seen the data, let me add a note to Rands comments about affiliate links. There's a big difference between what he's talking about and some of the stuff that I've seen regarding paid links.
If you think you can buy 10,000 paid links and get top rankings, I'm afraid that you'd be surprised at how easy it is to see these things. On more than one occasion doing demos at the booth I ran a basic comparison report and immediately spotted the people doing this. And this report is available to all members, for free (just think about what you can get if you're PRO!)
If you see the top nine people only have a few hundred external links in our index, but number ten has 10,000 external links, and a mozRank which is out of whack, you can guess what's going on. An advanced link intel will give you the specific examples.
In my opinion, this underlines how important deep link intelligence (rather than simple link counts) is. We'll continue to work on identifying these algorithmically. You can bet the engines are already there.
Perhaps as you guys continue working on trying to identify paid links algorithmically, you could find a way to notate these links (with a tiny icon, as when a link appears in an image, for example) as being "possibly paid."
Keep up the great work!
moved to Linkscape feedback thread so we don't lose this great feature request :)
Without jumping on the bandwagon one way or another, this is a good example of why some people might want to block Linkscape from crawling any sites they own - or their preference to having their website unlisted from the Linkscape index all together.
Not me mind you.
But others who don't want it to be so easy for their link profile to be discovered, analyzed and possibly reported.
I'm glad they addressed the false positives problem, although they seem to have ignored the fact that "false positive" also includes people who don't know better. I hear from scores of people through Q&A who've had a site disappear completely for no apparent reason. Unless they're all liars (they're not) their sites either came up a false positive, or they did something "bad" without realising it. From my experience in Q&A, it seems that Live does the best job with giving feedback, but that reinclusion / reconsideration in all engines is often a nightmare even (or especially) if you've not sure what you've done wrong, or if you even did anything wrong to begin with.
Very interesting about sitemaps. I shall take this and run with it too ;)
In addition to our Q&A system, you can also get engaged with the webmaster support forums that the engines have. For instance, try Google's webmaster help forum. As long as you keep a cool head, and can provide some specifics, they're very helpful.
I dont know the whole thing sounds fishy to me about the sitemaps. To tell Google, these pages are important and these not is like taking your own kid to prison to me. That is what nofollows are to tell google what page I dont care about not a sitemap. Just my 2 cents.
I was surprised to hear the affiliate link comment too. Makes me wonder how they are defining a product/affiliate of high quality.
Are they using the same standards as regular links or have they decided (because of how close these are to paid links) to deem them worthy through another set of standards.
This is my exact question (re: how they are defining product/affiliate of high quality).
Obviously we'll never know exactly how they're making teh determination but one can't help but wonder. Seems a very subjective decision to make for such a granular bit of their work.
This is my exact question (re: how they are defining product/affiliate of high quality).
Obviously we'll never know exactly how they're making the determination but one can't help but wonder. Seems a very subjective decision to make for such a granular bit of their work.
"Put really important pages in your sitemap, rather than every page on your site."
This really changes the way to look at Sitemap from now on.
Does this also goes in for Google ?
Hey Rand,
Just to clarify, we set a pretty high bar on what's considered useful affiliate linking. Let me give you an example:
Take the (well read) strobist.blogspot.com site. David has a page (here https://strobist.blogspot.com/2006/04/strobist-bookshelf.html) which has links to a ton of really useful resources for the photographic community. All on-topic, and widely referenced in a lot of the material on his site. His affiliate id is in all of those amazon.com links, and he makes no secret of that (see here, for example: https://strobist.blogspot.com/2007/07/hot-list-q2-07.html). Importantly, the rest of his advertising (in the sidebar) is dutifully nofollowed, because he personally isn't endorsing any specific product there.
The point is that the links he references in the post are highly relevant, and his audience has repeatedly come back to those links as a resource. This I would call a good example of how an affiliate link can be a valuable addition to the webgraph.
Hi Aaron,
thanks for your comments.
> The point is that the links he references in the post are highly relevant, and his audience has repeatedly come back to those links as a resource.
and that's why they should be pass link value and should not get the webmaster who posted them on his website into trouble with the SE......but it should not matter if it is an affiliate link or not. If it does not meet the points that you just made, the links should be discounted, plain direct links as also aff. links.
If a page, site has too many of those links that are highly irrelevant and no audience will ever come back to them as a resource, it should be penalized
categorized as being potentially a FFA page or link-farm, which do not have affiliate links or for being a "thin affiliate site" or for being simply a completely worthless page that nobody in is clear mind would want to waste his time on.
> This I would call a good example of how an affiliate link can be a valuable addition to the webgraph.
And there are many more. This is the whole core of the issue. You always talk as if affiliate stuff is almost always spam with a few exception.
Which is like talking as if all SEO spam the SE using deceptive methods, but there are a few exception like Rand, Aaron and Danny and that's it. ... or like talking as if Googlers usually spend all their day at the Googleplex in the cafeteria or gym or playing role playing games, with a few exceptions, like Matt, Larry , Sergei and Marissa, who actually do real work and get something done over at Google.Both things are not true, but if people talk like that, it would make SEOs mad or you and other Googlers, wouldn't it?Well, I hope that your perception will change, now that you operate an affiliate network yourself, which would be like "aiding spam", if things were the way you think they are. ;)Cheers!Carsten
@Rand for the sitemaps #5 are you referring to XML sitemaps or html sitemaps? Sorry if that was a given and I just missed it.
As far as that is concerned, for one of our clients, the sitemap is by far the second most important page on the site.
Low Exit Rates, good presentation of navigation options. Sitemaps rule.
XML and Txt is what they mentioned in the panel. I'd make sure for SEs there is an XML and an HTML for users.
I think XML might give you a little more explicit control. But from my experience, and the messaging I've heard, is that the engines and and will use both. And if you don't have an XML sitemap, they'll get a lot of good info from the HTML one.
Im sure they are referring to xml sitemaps as the real purpose of this was to let G know which pages are important yet most people still add every page including the terms of service type pages etc. This is just wasted time for Google looking at those urls again.
It obviously makes sense to have both with a full one on site for sure.
Especially that sometimes the sitemap included in the site itself is not as complete as the one in the XML form, where you can point out to the priority of a page compare to the other ones. This type of information is only useful to SEs
Yeah, for one of my clients, Sitemap page is the 2nd most Important and ranking page for most of his keywords other than the index page. This was not at all intended.
So here's my .02 on sitemaps...we are getting conflicting info. While the search engines say use them, the SEO panel on ask the SEO's pretty much all said they don't use them. The Panel included oilman, bruce clay, sugarrae, and many others I listen to. Then the search engines come up and say use them.
This reminds me of the Google posting about not worrying about re-writing URLs - sure that might be the official statement, but every SEO knows that it does help.
I'm getting a bit confused, but for now I', listening more to the SEOs than the engine, but that is just me.
I still use them religiously. I was not about to mess with a panel like that, I have huge amounts of respect for them all, but I have seen good things from having sitemaps.
One, which Eric Lander used for his panel, is that I use them to influence site links. I can't promise it works, but I tried it with my company and I saw results.
So since it takes very little time and can have such a great impact I do continue to use them, submit them, and reference them in robots.txt.
I think most of the top SEOs don't use them because they can mask the discovery of other issues.
For instance if you don't have a site map and you do a "site:" query and it only shows about 20% of your pages are indexed, you know you have a few issues on your hands (not enough link juice flowing into your deeper pages is probably at the top of that list along with some navigation issues).
But if you submit a site map, the search engines will try to follow that and will likely index all of your pages and thereby not give you the insight to potential problems.
So everyone has to figure out what their immediate goal is.
The goal for most of my sites is to get all of the pages indexed - so I always submit xml site maps when I can.
On some of my sites I also try to include a shortened version of my html sitemap on any 404 pages for the rare human visitor.
I always compile the HTML sitemap.
I never upload an XML or TXT sitemap.
Sounds like it was a great session. Wish I could have been in that one, but unfortunately there is only one of me and I wanted to check out the In-House session :(
Thanks for the review/update. Bottom line still seems to be good content and lots of relavent links.
Awesome information, lots of good stuff especially about the PR sculpting and affiliate link. I was really shocked about the response to both.
It is really true that adding a sitemap on search engines will increase your site's organic traffic. But i have had problems with msn and yahoo about the sitemap. :(
rand, thank you for sharing this, it is interesting to read about the search engineers's opinions/views about affiliate links
Just adding my thanks to the long list. It's comforting to see that there's generally a consensus between the different search engines.
Thanks for this update Rand.
See, that wasn't too hard (regarding the affiliate links thingy). The answers from the search engines are actually something you can work with...
... no need for rumors, suspicions, fear and all that nasty stuff that was caused by ambiguous statements made in the past.
I am getting confused after reading this post. What i always thought was sitemap was for getting every page indexed on your website ,not for the most imp ones as most of the time most imp ones will get indexed automatically.
I would imagine that something like Pagerank could be used to provide a rough filter for what qualifies as a useful affiliate site - an affiliate link from an unranked page may have no impact at all, as opposed to an almost-zero effect for a non-affiliate link from a similar low-value page.
However, this does open up the use of an affiliate scheme as an SEO tactic, as well as a traffic/marketing source. If affiliates can help your site's rankings as well as sell your products, then it becomes highly lucrative to set up an affiliate program on both counts.
In reference to number 2) - I assume that this is only relevant to affiliate programs that track using their own domains like amazon. If they use a network, I assume there is no juice to pass since the tracking links are redirected through network owned domains.
Furthermore, some top affiliate sites use server side redirects before sending the traffic through the affiliate links - wouldn’t this prevent juice getting passed even through programs like amazon?
Great job in summing up a lot of points many SEO experts miss. You are right in saying that links were the one of the most important things to build when optimizing.
Great information here. Especially eye opening was the sitemap subject. As someone who is new to this whole thing it's an issue I hadn't given much though too. Will definitely do some more in depth investigation before next project begins.
Regardless of where you fall on issues like these, they are at the least eye opening to new areas of concern, or new perspectives on things you've already decided.
Great Post. Thank you for the information
Thanks for the roundup, Rand. Good stuff.
Fantastically concise round up - loads of very actionable stuff there - thank you!
I'd find it interesting to know how they deal with reports on paid links. Google says in its guidelines that one should report those links and they'll handle it. but I'd be interested if it really does affect rankings in a given period of time (say a month or two).
I think it's unfair if you strictly go with google's guidelines and see your competitors obviously breaking the guidelines and (therefore) rank above you in the SERPs.
If google said that they only penalise too obvious and big sites ... well that would certainly change something :-)
You should always make the right decision for yourself. That said, having waded through some data, I'm inclined to think that the engines are discounting obviously shady links. I bet "shady" is defined as the Supreme Court defines "pornography": I can't define it, but I know it when I see it.
If you think you're getting outranked by shady means, you too can look at the data. Go back to your own content and tighten that up. And try the free basic comparison report which shares external link counts. So far I've seen quality content and relevant links out ranking the shady ones.
I rather put all my site to sitemap, all 3000 pages or so. There is many theories about sitemap sizes, in the reality, they can be huge. Sitemap improves enormously robots to find pages and avoids meaningless try-fail crawls.
One thing about size I've heard from at least Live and Google: they have to download that sucker. I've heard horror stories about sitemaps that are many megabytes in size which certainly makes the engines think twice about how frequently they want to get that stuff. Think about it, it hurts your bandwidth and theirs, and they need to get millions of these things.
That said, I've heard positive comments from the engines about having many URLs for completeness, and having few URLs to ensure the most important stuff gets crawled.
There is an absolute value on how big they can be, isn't it?
Sitemap.org states 50.000 urls or a max of 10mb (10,485,760 bytes)
I´m personaly abit puzzled by the information that you should rather put your most imporant pages in your sitemap.xml rather then your entire site, which to me is abit unlogical as the sitemap.xml was first invented to ensure full indexation of your site.
As of the entire post, great stuff Rand and alot of very intressting perspectives this creates, specially the affiliate information.
"the practice of using nofollow to flow link juice"
I thought nofollows werent used to pass link juice. Quite the opposite of what i thought..
nofollow prevents the flow of link juice from that particular link but using nofollow on a few of your internal links can help you control where you want to focus your site's link juice.
We see a lot of nofollow usage on links, including internal links. I think Rand already released some stats in our Linkscape announcement post.
We've seen success from a broad and carefully planned PR sculpting effort. The engine reps suggest that you should focus on other things first, such as developing high-quality content that is intrinsically rank-able. I don't see how I can disagree. If the content isn't rank-able on SERPs you care about beforehand, pumping it up with links won't drive you converting traffic.
Yeah also preventing the unwanted pages from getting indexed or ranked.
all this talk about linkscape and still no time to type an ip address and user-agent.
Give it up for a bit. The SEOmoz crew is not the kind to hold back info just to piss people off. Let them get things together. They are webmasters too.
As I've said in a few other comments around the web and here on SEOmoz, it's something we're working on, and you can certainly expect a method for exclusion by November (remember, our index only updates once a month). There's a bit of internal discussion on how to make this happen, but I'm returning to Seattle tonight or tomorrow and we'll be meeting ASAP to address.
Thanks Rand for a detailed report.
@mozers thanks for a lively interaction after. Always enlightening.
@Nick awesome work, "we're not worthy."
actually, it is going to be tough to top Linkscape.
I posted a quick question over on BruceClay.com but it's likely in a queue.
Over on their coverage they stated:
Aaron says that he's involved in trying to get rid of spam. He hears a lot about companies wanting to put up different versions of content for different countries. They wonder if it's going to be a duplicate content issue. He says that if the URL and the path to the content is reported to Google as specific for a certain location, Google won't see it as duplicate content.
@Rand - did you hear this, and did you understand that if, for example, you publish dupe content on multiple pages under one domain, but then geotarget those pages at different geolocales that Google won't filer?
I know that the filtering rules can use geotargeting (ccTLD etc.) to determine different versions of the truth (i.e. different domains) based on different Google search properties, but I've seen first-hand how dupe content under one domain, but under separate geo-targeted sub-folders causes havoc for the filters.
Any ideas/feedback on this is appreciated in advance :)
why would it cause havoc?
If is a subfolder or domain and has its unique document name I dont see why there would be a problem.
It should always return the 'main' page for that content with the duplicate content filtered, in a geo targeted result it should clearly provide that reletive document no? Then it simply filters your main page out based on instructions.
Love the "Search Engineers" terminology, that's what I'll put on my next business card :)
Great post, Rand - Thanks!
This kind of info, straight from the horses mouth, is very reassuring to my strategies, thanks again.
I could be crazy... but I'm pretty sure that Google owns Webmaster Tools.. not Microsoft? (number 3)
Microsoft's Live search engine offers a service called "Webmaster Tools" at the link I pointed to - I think it's just a common name for the offerings :-)