I don't generally like to pick on Google, or anyone else in particular for that matter, but since the search giant is hitting on all cylinders, I think they can probably take a little nudge or two. I'm going to keep this short, sweet and to the point (and also very self-centric, because it's tough to effectively analyze everything that might have happened on someone else's site):
- A very popular post I wrote a few weeks back called "Google PageRank Losses for Hundreds of Websites" still hasn't been indexed. Yahoo! has it. Live has it. Google's visited it with their spider plenty... What gives? There's almost 1000 links pointing to it.
- Hey look... We're back to ranking #2 for "recommended list" after we 301'd our old page to the Marketplace section. I'm going to go with the theory that we got penalized for having too much content in CSS hidden layers because we used drop-downs for the list of companies so the page wouldn't be huge. I'm trying to make it good for users, Google! Don't penalize for that!
- SEOmoz's Toolbar PageRank on our homepage dropped from a 7 to a 6. I had assumed that because it occurred during the normal PR update, and not 2 days beforehand when paid link sites were hit, it was a natural drop. However, logging into Webmaster Tools today, I note that the homepage has gained almost 10,000 new inbound links since I last checked a few months back... Seems awful fishy.
- Speaking of Webmaster Central Tools - what the heck is this?
Thanks to the comments, I realize how dumb I must have looked for this one. It's actually just a script from Joost de Valk, available here :)
- Last thing on Webmaster Central - in that links section, why not offer a "sort by" option? I know I can download to Excel, but it's such a comparative pain, and it seems like the kind of thing that would take a programmer 20 minutes to add. I've heard lots of folks ask for it...
- Blogsearch is such a joke! I can find three copies of our content on junky splogs that no one ever links to within 10 seconds of posting a new entry on SEOmoz, but even 2 or 3 days after publishing content here, blogsearch still hasn't picked it up. Yet, supposedly, according to Google's own reader stats, we're one of the top 50 or so feeds on the web (Techcrunch's list).
- If you were serious about removing paid links from passing value, this page wouldn't rank anywhere for the term "SEO." Take a look at who is linking to the domain and the page and why. There's so many dozens of visible examples of this "paid network" sort of effect on the web, it's hard to believe no one's written an exposé yet (technically, SEOChat is doing the same thing, but they've been in the top spot for so many years they've earned lots of natural link goodness, too). You know who really should be ranking there? This guy. Yahoo! and Live both have him in the top 10 :)
- After Dave Naylor wrote about this ".com" problem and this search results problem, I figured you'd take the time to figure out what's going on with it, but wow, still happening. One of those issues is 6 months old!
- This is a relevance problem
- This is a relevance problem
- This is a relevance problem (and a potential copyright violation, but we'll let Sarah worry about that)
- If there are 2.43 million results for this query, why show only up to result 893, even with &filter=0?
OK, I've got lots more, but that's probably enough for tonight. I do, however, have one request for the readers here, though, which is to please go ahead and add your personal gripes on sites you can share in the comments. I think it would be tremendously valuable to see where common issues occur.
I can tackle a few without breaking open my debugging tools:
- the PR displays in your backlinks are a custom script from Joost that you installed. It's not our fault if that Firefox addon is acting up on you. :)
- we did do a full PageRank update several weeks ago. seomoz.org went down one notch not because of link selling but just because there's less PageRank flowing around in some areas (e.g. search and SEO). Vanessa Fox's site dropped by one as well, and for her as well, it's just a case where less PageRank is flowing in some niches of the net. PageRank doesn't always monotonically increase.
- I see https://www.seomoz.org/blog/complaining-about-some-google-missteps indexed now. We didn't do anything manual to keep that post from being indexed before, just like we didn't do anything manual to get it indexed now. I may check it out at work, but the answer most likely lies in how much you linked to that post from internal pages.
- We also didn't do anything special/manual for your recommended list or how that page got scored. I suspect that 301'ing the old page was helpful, but we never guarantee that we're going to rank a given page for a given query.
- I'll pass on the "Sort By" feature request, but when you're only displaying a small fraction of your links on the web page, it didn't seem that helpful to me to sort that small fraction of links. That's why the export feature is so handy--so you can slice and dice the data yourself.
- Blogsearch does work on pings. Lots of sites will automatically ping for you (e.g. blogspot), and lots of software can ping for you as well (e.g. wordpress). I know that one other search engine wasn't getting pings to Google for a few days because of an issue at the other search engine, but I believe that issue was fixed a week or so ago. I'm guessing that you've customized your software quite a bit--which places do you ping?
- To say "page A shouldn't rank at this postion because it has paid links" commits the logical fallacy that it's the paid links that cause page A to rank, as opposed to the other links to page A.
- I'll be happy to pass on the relevance suggestions, but [randfish seomoz] does return lots of randfish/seomoz related material. :)
- I got a full 991 to 1000 results for your last query, but things can vary slightly depending on which data center you hit, if we're swapping in new data, etc.
"- we did do a full PageRank update several weeks ago. seomoz.org went down one notch not because of link selling but just because there's less PageRank flowing around in some areas (e.g. search and SEO). Vanessa Fox's site dropped by one as well, and for her as well, it's just a case where less PageRank is flowing in some niches of the net. PageRank doesn't always monotonically increase."
This speaks volumes and I like it!
At my local Border's bookstore the IT section used to take up almost one entire corner of the store - about 1/9th of the total floor space of a somewhat huge store.
However now, as a percentage of the book-buying population, less people are interested in IT now that the dot com boom has come and gone (or they are buying less books for it) and now the IT section has been relegated to a section about half of its original size.
So PageRank is like this as well.
Links to search related blogs and websites used to be pretty high as a percentage of overall links on the web. But now, less people outside of SEO/search are linking into SEO/search related sites. So even if a site gets more inbound links (10,000+) - overall the inbound links to the website (and the referring SEO-related sites) are down compared to the rest of the links on the web.
Matt well done! And you didn't once use the phrase "Stochasticity Adjustment".
vingold, I did throw in "monotonically" for Hamlet, but otherwise I try to avoid being too math-y. :)
Matt - thanks for going above and beyond the call of duty. Much appreciated.
The only one I'll still quibble with is the recommended list. Two weeks ago, you could type in the exact title tag and add site:seomoz.org, and it would still be the very last result - it was a clear penalty issue. The 301 made an instanteous difference once it was indexed and the result now ranks for all sorts of good stuff, as it used to prior to our modifications to the display.
Blogsearch pings... Hmm.... I should really connect with Jeff & Mel. Yes - SEOmoz is completely custom built, no CMS software, so it's very possible that we're not pinging at all.
BTW - Si's wondering if you liked his post on PageRank. I told him he'd have to ask you in person :)
Rand, I think you might have some issues of regarding seomoz.org & seomoz.com.
Take this wget:
wget https://www.seomoz.com/blog/toolbar-pagerank-losses-for-hundreds-of-websites--09:28:33-- https://www.seomoz.com/blog/toolbar-pagerank-losses-for-hundreds-of-websites => `toolbar-pagerank-losses-for-hundreds-of-websites'Resolving www.seomoz.com... 209.160.33.125Connecting to www.seomoz.com|209.160.33.125|:80... connected.HTTP request sent, awaiting response... 301 Moved PermanentlyLocation: https://www.seomoz.org/blog/toolbar-pagerank-losses-for-hundreds-of-websites [following]--09:32:41-- https://www.seomoz.org/blog/toolbar-pagerank-losses-for-hundreds-of-websites
When I did a wget from the Googleplex, I eventually got a 301 from the seomoz.com url to the seomoz.org url. But look at the timestamps: " --09:28:33-- " was the initial fetch and "--09:32:41--" was when the 301 came over the wire. Assuming that I'm reading right, that means almost a four minute delay on getting the 301 from seomoz.com to seomoz.org. Googlebot will wait around for several seconds for a page, but it won't wait four minutes. Instead, the connection will time out and we'll treat those urls as separate (and think that we couldn't fetch the seomoz.com url). So if a bunch of people are linking to your article, and some link to seomoz.org and some link to seomoz.com, that PageRank is getting split between two urls, and the long delay on the 301 response can cause Google to believe that the urls are separate and therefore cause dupe issues.
From doing a cursory look, I think something like that might be your issue and could be causing many of these ranking issues for you. On the bright side, that also means that it's in your power to make things better quickly. I'd check on your latency on serving 301s from seomoz.com to seomoz.org, and if things are taking a really really long time (e.g. four minutes), try to get that latency down.
I believe that will help with several of the queries you asked about.
Since we are on the topic of google issues and the .org/.com of SEOmoz, I thought I would mention that SEOmoz is lucky with their real estate on a search for seomoz.
I wonder if this 301 latency issue is also to blame for the ability to rank the same page for both .com and .org at the number 2 and 3 spots?
Queue the flood of "so why does my site..." emails to Matt for taking this much time and effort to address these questions.
It is appreciated by all of us to read and remind us that there are other factors such as server issues and latency that may be involved (other than penalties and conspiracies, of course). ;)
Oh my god - I was thinking the exact same thing:
Come one come all! Today Only! Online Site Review Clinic by Matt Cutts! Only at SEOmoz.org!
Wa wa wee woo! I would never have guessed that the mozzers would overlook such an important issue.
I was thinking the same thing as Matt that you might not be pinging b/c you weren't showing up in blogsearch and ANYONE can show up in blogsearch.
Wow. Detailed reply. If the whole set of SEO sites is seeing a (toolbar) PR drop as a result of the adjustment, that presumably means it is factoring into an iterative calculation somewhere in order for the total effect to propagate around to people who 'done no wrong'.
In other words toolbar PR and PR are both calculated by their own separate algorithms (or toolbar PR is actually going to mean something again).
Until I read this, I was hypothesising that the toolbar PR penalty was an after-the-fact penalty whereby the calculation was done and *then* reduced as a penalty whereas this looks like the penalty was added earlier in the process which has all kinds of potential knock-on effects. Interesting!
I really need to finish my post on stochastic convergence. Sorry Hamlet!
Whoa folks. I need a moment here. If this was facebook, I'd high-five everyone. (or maybe throw a cow at everyone?)
This former litigator feels like she just came from the battlefield into Shangri-law.
I'm new here, and I didn't really appreciate how mozzy the industry really is until now. I think its just fantastic that the SEO/Ms and the googlers can be friends.
Thank you Matt for responding so respectfully, immediately, and fully to Rand's post. I'm sure you have other things to do with your morning.
I just assumed you guys had an awkward and strained professional relationship based on mutual distrust and co-dependence. And maybe that's true, but I'm glad to see that it doesn't prevent everyone from having meaningful conversation about common interests.
Lawyers have a lot to learn about collegiality from search professionals.
Thumbs all around.
i think the one great thing about Google is that they listen. Matt alone is responsible for getting a lot of things fixed on their end. They're not perfect but what search engine, or person, is?
Rand is very good at bridging the gap without offending people. Some might say he's a bit too soft and political but I think that's his niche. There are plenty of critics but Rand is one guy that can walk both sides and that is his unique strength among the prominent SEOs.
Hey Sarah Bird, and congrats on joining SEOMoz! SEO discussions aren't always cordial everywhere across the web, but Rand sets a pretty good precedent of politeness on SEOMoz. :)
Tom high fives sarah!
Sarah throws a cow at Tom_C
Again - I say Awesome!
Thanks for the WELL detailed reply, God this is soo great !
Matt
Thanks for the excellent replies. This is more information than we usually hope to pry loose at Pubcon. The only problem is I am going to lose a lot of sleep trying to figure out this comment:
On second thought, the URL Rand is asking about, the one that isn't ranked, it is this one, which still isn't indexed. No conspiracy theory here, but it does seem odd.
https://www.seomoz.org/blog/toolbar-pagerank-losses-for-hundreds-of-websites
Jonah, looking back I was checking on this post rather than the one that Rand was asking about. But all my comments about nothing manual being in effect still apply, and from looking into it, the strange seomoz.com vs. seomoz.org thing appears to be the root cause.
Rand, please tell Si that I did enjoy his PageRank article very much. But that I'm also still just a little hurt that he said Yahoo searches stacked up fine for him compared to Google. :) I'd encourage Si to try iGoogle as his home page for a while--no need to see that purple Yahoo page too often. ;)
Matt
I never suspected any manual intervention of a SEOMoz page because it is unflattering for Google. I have seen plenty of antagonistic pages indexed. I should have highlighted the part I am intrigued by, but the answer most likely lies in how much you linked to that post from internal pages.
The thing is, if is has something to do with how Rand is linking internally to the post, as you suggested, I am completely confused. Yahoo backling data suggests that the post has a total of 908 links and 685 from outside and 220 or so from SEOMoz.
Do you stand by that suggestion?
If you do, since Rand doesn't appear to have canonical problems, duplciate content or internal Nofollows in place, then I read between the lines that too many internal links are BAD for a page. While we all know that a sitewide link to an external page may be a bad signal (and/or suggestive of a paid link), I would assume that sitewide linking to an internal page is a good quality signal.
Care to clarify and give us all something to speculate about?
Vanessa lost her sitewide links from Google blogs since the last update, and otherwise probably has 1/100 the number of links as SEOmoz - the scaling just doesn't seem right, Vanessa's sources for links are probably very similar, especially considering how many theme developers suddenly popped up to PR7 and at least one typeface blog.
Those niches are not so unconnected from the search related blogs.
SearchEngineLand SEJ and Marketing Pilgrim also a PR7 but have the all powerful link from the Google blog.
In fact a large number of Googlers have a PR7 most likely due to that one sitewide link.
Matt, I brought this up ago over at Webmasterworld. I'm getting some pretty significant pruning of searches.
https://www.webmasterworld.com/google/3480023.htm
Someone who will remain anonymous said to me today:
The only thing I'd wish for is some better anchor text on that link :P
This is the problem when I get no sleep the night before, then feel compelled to blog at midnight anyway... I think I need to get more people contributing to the blog - it's too much for just me :)
You know were to reach me Rand :)
No problem buddy , It happens to all of us, Although I did chuckle when he corrected you, Not cause I was being mean, Just because stuff like that has happened to me before :)
Rand,
I'm looking for a new job and willing to relocate.
Plus I had a dream last night that I had my very own nifty SEOmoz collector's card. It was sweet.
Maybe you need a vacation. Someplace warm. Where the beer flows like wine. Just a thought.
A great and enlightening discussion here, but I still have to confirm one point - I assume Jonah Stein's question about internal linking remained unanswered, so I decided to ask again.
Are there some clear recommendations as for internal linking patterns? We are all pretty sure that external site-wide links are potentially dangerous, and what about internal linking? A really interesting point and I would eagerly dig it deeper.
Hi Ann, the more I dug into it, the more I lean toward thinking that the seomoz.org vs. seomoz.com factor is the larger/more important issue rather than internal linking. If the .com always does a 301 to the .org, then for [seomoz] you would not expect to see both the .org and the .com showing up, for example.
If Rand were doing something intricate with 301s between seomoz.com and seomoz.org, I would advise going with the KISS method (Keep it simple, silly), because I think that four minute delay could be causing issues. Once that is resolved and Google treats the .org/.com well, then I'd expect rankings to do better for the specific searches Rand mentioned.
Jeff & Mel looked into this, and we can't find the same problem you see from the 'plex. All our requests are resolved in a few seconds. We're not doing anything fancy - it's a direct 301 and both domains are coming from the same DNS host.
Obviously, you got the 4 minute delay, but we can't reproduce it - has anyone else from an IP address we can track made a ping request to SEOmoz.com that doesn't resolve instantaneously?
Rand, I been having some delays in accessing SeoMoz.org, from Japan.
Could it be that the Moz platform is data hungry, and restricting delivery of content, because of bandwith aloction, or server computing resources, that would explain a 4 minute delay in serving a page.
If Googlebot came to the page, and it was delayed in delivery, it will not index it.
This problem has simularatieswith having MySql connectn unavailable, and you get error report text in Google snipet.
I would think the problom page will be indexed on the next Googlebot atempt if the delivery is not hindered.
But if Googlebot noted that the page cannot be accessed it will lower its priority in indexing.
Google has two robots, one indexis all the time if the site is very popular, and another twice a month, or less depending on popularity.
So the young and the fast Googlebot, may not come to this page, untill the stedy and slow Googlebot will index it first.
You can actually force the grandpa to come earlier, if you do a url removal from the Webmaster Consol, and then a few day later cancel the removal.
I forgot the name of the regular Googlebot, it has a special name, I seen it in my raw server logs.
Maybe Matt can provide us the name so we can keep an eye out for it...
It is really fast, it comes like a turnedo, indexing everything at once. Something Java agent...
Maybe related, but do you also 301 the www.seomoz.org to the non-www seomoz.org version?
Hi Rand - it was happening for me too. I've dropped you a mail, hope it helps you debugg the problem!
I'm on the East Coast - and I've been having connection issues to seomoz.org for about two days now.
I was thinking it was because you were fixing some of the problems Matt referenced, or because you were upgrading the site with the new private messages feature.
It would be interesting to know if these latency issues were new (I've never had a problem before Wednesday) and if they are new (last two weeks or so) then they can't be assumed to be causing the problems Matt is detailing.
Yes, I had long load times as well for a couple days around the sametime Vingold did. (just as an FYI incase you can do anything useful with the data)
Don't hold back Rand :-)
Those webmaster screenshots with the bold PR next to the links.. Isn't that a firefox/greasemonkey plugin (and hence not google's fault)? We don't get those bold PR ratings next to the links in webmaster central in the UK - maybe it's a US thing....
My pet peeve is about how google is really bad at determining when IT in a search string is related to computer stuff and when it's not. I wrote a post ages ago but then I can hardly expect Cutts himself to swing by and read it ;-)
Post here about the difference between it and IT
Also - do you think that the page not being indexed might be something to do with this page? (these comment links are nofollowed right?!) Surely with the amount of scrapers out there Google will get it wrong eventually with the duplicate content thing?
I mean it's not ideal but I think to highlight the fact that one specific page isn't indexed by Google but is for Yahoo/Live is kind of missing the point. I imagine in the long run Yahoo/Live get it wrong more than google does with all these scrapers and you'd need to look at lots of pages together rather than picking the one specific post which Google tripped up on.
Also - what are you pointing out with this search? Here in the UK I see you no2 but don't see any potential copyright infringements?
Ha. Good job you mentioned that difference between it and IT - I "noticed" that again yesterday and was going to blog about it! I had forgotten you already had...
I might still write my post because it has some interesting differences between paid and natural searches, but I'll make sure I remember you've already written about it!
I don't think its a temper tantrum at all. I have a few of these indexing issues myself and its nice to know it can happen to some of the big guys as well.
That tells me its not just me - its Google.
When someone does a post like this, I think you'll find dozens of people come out and say "Oh wait, yeah I have that too..." - and then we reach critical mass and Google does something. Hopefully.
I didn't see it as a temper tantrum, rather just pointing out a few oddities w. Google. I still think that by and large their SERPs tend to more relevant than the other three engines, although I have been quite impresssed by Live lately.
My "peeve" is just related to a problem I am having with one of my client's sites. I'd like an easy way in webmaster tools to tell the search engine when you are 301'ing an entire site. Or to know if you are under some kind of penalty or waiting period. I tried to follow best practices, submitting a sitemap for both sites and then changing the old one to show the new URLs, but this didn't seem to work. Obviously all of the page URLs on the old site are 301'ed using htaccess.
My client was ranking #1 for a bunch of locally competitive terms, and now he is #50, with BOTH sites appearing in the index. I understand that it may take awhile to re-accumulate rankings, but the fact that both are appearing in the index worries me. I think Google thinks I am doing something fishy, when I could assure them it is entirely white hat!
Wow, lot's of banter from Matt, and it's very much appreciated and welcome!
I'm led to wonder if there's a facial hair club, kinda like the Masons, that has sparked this level of interaction from Matt on Rand's post? Do you guys have a funny handshake or what?
Thanks for the input, very useful.
Rand, I believe the funky colors and strange PR numbers aren't coming from Google but rather from a script you have installed in your browser (probably Joost's https://www.joostdevalk.nl/code/greasemonkey/gwt-external-links/ ). Can you check to see if you have it installed? You can turn Greasemonkey off and just reload the page - it should only list the links.
To John: It IS my script indeed :)
To Rand: it's -1 because it can't get PageRank (yeah I know, bad usability).
The orange striked through links are nofollowed links, those you CAN complain about, because I don't know why Google would want to show those, unless they use them in ranking somehow...
The explanation (and installation) of the scripts colors and stuff is here. I guess this means that Google will have to either integrate my stuff into their pages, or email me when they do code changes :)
Great job, Joost! I was going to login to webmaster central to check this fantastic new functionality! :)
Cheers
Me too. That's an excellent function! Hurray for randomly stumbling across helpful tools.
Thx Hamlet :)
Joost, do you mind if I transfer this bug report over to you then? :)
Hehe well Matt, only if you promise to give me three weeks notice every next time you'll change the interface ;)
You could ofcourse also just make this data show up in Google Webmaster Tools yourselves, that would be a lot easier on your servers too I guess :P
Hey Matt, in the latest release of my script I've fixed the bugs Rand showed, I dare you to fix your bugs just as fast ;)
you can send the bug report to me, just include the server's root access password pls
What a great liitle greasemonkey script :) Any plans to do the same for yahoo site explorer - since they do a much better job of listing who is really linking to you... JK.
I might... If only I had a lot of time... :P
Hey -
I just met you last night Via email ( I asked about your script ) , but you seem like a cool guy -
Make some time, :) Its a perfect way to get more readers, and now you just got exposed eevn more, :)
Ey bestwebsites, throw me an email, i'm working on the yahoo version and need some beta testers :)
Great tool can't wait to try it!
I think this whole excercise shows the difficulty in looking at sites from the outside to see what potential problems they have. Being that it's Rand I wouldn't have thought to see if his 301 is really a 301 and if it takes 4 minutes to resolve, the same with the blog pinging, it's just assumed that he was doing that.
As far as Matt seeing all the search results shown in the count, it's a little like taking your car to the mechanic and telling him that it makes a noise when I turn left over 30 MPH. He drives it around for half an hour and doesn't hear it, giving you a look like your a complete idiot. If it helps, when I read Rands post last night I did follow the search link, and did go to the last page. I don't remember the actual number but it was around 800 results.
I just check it again, clicking the "more similar results" gives you the 1000 results.
Nice to know that my site didn't suffer any PageRank penalty, 6-4 does seem particularly harsh though, I have one of them pages Rand:
seoco.co.uk/optimised-web-design.html
Iv'e tried changing the page title, changing internal anchor text, adding external links with random anchor text, you name it. I am getting a redesign soon and that sucker will not be getting 301'd anywhere.
Not to sound like a valley surfer but: This post if friggin' awesome!
Not only did Rand point out a ton of things that personally piss me off as well but Matt is here addressing all the questions. Thank you Matt!
I wish I could get this level of attention to my own site but I'll take what I can get. The back and forth exchange is extremely valuable.
I was going to do some of the same AJAX style hidden content for my site redesign but I got the same worries as Rand and stopped. I think it's really too risky at the moment. Too bad, it was going to be a funky site.
Anyway, thank you Rand for writing this and Matt for responding so thoroughly. I think the comments are actually more valuable than the article. Nice score on the personal site review too Rand ;)
Matt, is there any technology you'd suggest for hiding content to make a cleaner page without triggering the spam penalty?
nicknick, these posts about content and Flash are related:
https://googlewebmastercentral.blogspot.com/2007/07/best-uses-of-flash.html
https://www.seroundtable.com/archives/014209.html
In general, you want the text to be exactly the same as in the Flash or AJAX; otherwise you're into the Very Risky areas. :)
Hi Matt,
Thanks. I have always followed the best practices for Flash use and my site ranked when everyone screamed that there is no way to get a flash site to do well on search engines. Even though I did some questionable things, I always gave Google the benefit of the doubt that if the text was the same, it would know that it wasn't spam. I was right and so were you guys.
My question has more to do with content that needs to be hidden with CSS until such time as javascript reveals it. Like with drop down arrows on a page. Used to make the page shorter and more scannable but the content is in a hidden DIV. Is there any way to do this without incurring a penalty? Can we notify Google that this is content and not a trick to sapm?
I can imagine that this is extremely difficult from an algorithm standpoint but if there is an official best practice, I'd love to know.
And thank you for being so active here and providing needed feedback. You're an example to your industry.
nicknick, I think we do a pretty good job of handling that case (e.g. menus that get exposed when you mouseover text). I'd make sure that the menu isn't stuffed with tons of keywords, and really hard to find the mouseover location.
Hi Guys,
Thanks to both of you for the answers. We've never had problems with menus because they're generally in lists so I can see how that could easily be identified by a search engine. My main concern was with large blocks of text as Rand mentions. I still can't see how it could be differentiated from spam so I'm going to go with Rand's cautious approach on this one also. I'd hate to have a clean and funky site with all kinds of great effects but have nobody find the damn thing. I can't really fault Google for this because it must be a huge undertaking. If I were a spammer, I'd jump all over that technique to stuff loads of text onto a page.
Nick - this is one area where I'd disagree with Matt a little. I'd probably urge you not to put a lot of content into CSS "hidden" layers or attributes, as it can cause penalties (like the one I noted above). If you want to do drop-down, I'd almost urge you to use something like an iFrame to display the content or get creative with the CSS styling and naming conventions. The example I noted above isn't the only one that experienced this problem.
BTW - Not directly contradicting Matt - he's right when he says that Google generally does a good job with this, but I think you need to be even more cautious around this issue than he's advocating.
Rand, there is a method that is called unobtrusive JavaScript Dom, that takes into acordince of all user agnet when serving JS.
Or I would even use user agent server side script configuration for serving menu links to Google and other SERP bots, no need to feed JS to SERP bots because eachone will handle it differently....
Some parse one DOM element while get stuck with others...
So putting <noscript> atribute is not enough!
Excellent point, Igor. I love it when you contribute content here. Big thumbs up :)
Nicknick,
I'm not exactly sure of your dropdown needs, but I've been using Suckerfish Dropdowns for navigation, which use the :hover class, on a number of sites & this does not seem to be a problem for Google. Their algo seems to be pretty good at recognizing that this is actually a GOOD user experience and not some sort of cloaking. I think perhaps using things like "menu" and "nav" as the ID/class might be a good tip-off as to intent?
i have nothing useful to add here but i just had to say "wow!" this has got to be one of the most educational and insightful posts / comment threads i've read on here in a long time.
i had a 301 issue a couple of weeks ago that i luckily got cleared up quite easily and quickly. it's nice to know that even the big time experts run into the same problems now and again though. :)
lol I thought the same thing. When the comments become more informative than the blog post the people with Feed readers lose. It is a pity.
Agreed 100%. It's somehow comforting to know that even the experts don't have everything entirely under control at all times.
Wow, I'm a daily mozzer and I seem to have been the last to see all of the great comments from Matt. There is a wealth of knowledge here for those of us who understand how to extrapolate the underlying benefits from the discussion. I know the odds of Matt reading this comment are slim this late in the game, but I just have one thing to say to him. Google is becoming soooo powerful. Please help to keep it from becoming 'Evil' and falling into the wrong hands!
To Mozzers: This may sound cheesy, but it's way past my bed time. We must all work hard to make the Internet better. We must respect each other, Google, leaders in our industry and the amazing power of the Internet in it's entirety. I know greed can be a tough demon to fight off, but we must suppress it and work together to further the most effective form of communication EVER! (Internet)
Thank you Rand!
Thank you Matt!
Thank you fellow mozzers!
I'm happy to see you take on Google right where it matters, their search results.
I just hope that if you get some answers off-line that you'll share them with your readers and not pull the same link baiting stunt that some other SEO blogger has pulled recently.
With the next to worthless data they show inside webmaster's tools with all of the garbage nofollow links from bloggers that don't care to recognize their commenter's work, forum posts, etc I haven't looked at it in months. After I saw those PageRank and Anchor Text additions I actually logged in tonight. I'm glad John Mueller came along and explained what was going on because I thought I was getting some cheap-low-budget version of WMT.
The malformed search results count has been around for a while, I figured it was another famous "bad data push" but it sure seems too persistant for that, must be a glitch ...err feature.
Ironically - this same blog post is indexed 2 hours ago, following the google link above. :)
Things that make you go hmmm....
*claps* congrats rand, matt and all the other people that are making this post amazing. it's all very informative and i'm looking forward to seeing how comments continue throughout the rest of the day.
thanks to all of you great posters (and to rand who started it all). i've enjoyed reading it and a lot of questions i've had are being answered.
-sam
Rand
Given what Matt said about internal linking patterns being an issue for you (albeit for this page, not for the one that triggered the conversation), it would be great if you could diagram the linking pattern for this page and see is some conclusions become apparent.
Jonah
The first one might be due to a few reasons like it being tagged "Spamming & Black Hat". Or the usage of PageRank and PR too many times on the page. You're probably right on the tipping point; it's a percentages game these days with G.
Of course more along the lines of conspiracy we could assume things like the link to Andy Beard or shudder; Google censoring it.
I think your PR drop is most likely due to the fact that PR dropped for most sites, including Apple etc. It was across the board which I can remember having occurred one time earlier. My theory is that it's a kind of re-pageranking of the entire web to make up for the pages added. Plus, who worries about PR anyway? Everyone seems to always be saying it's dead, but the fixation still appears to be there....
[gulp]
Don't go spreading ideas like that, it is hard enough for people to give credit to a now PR4 ;)
I am still ranking for everything I ranked for before
Andy he is still crying, "Why G......!"
I'm so glad I'm reading this post a day after it was posted. Otherwise I wouldn't have gotten all of the useful information out of the comments. This thread would've been worth paying for as part of the premium membership! :-)
Great ! it's a good suggestions
The indexing bug is very puzzling indeed. Live has it as well. You search something, it tells you it has 1.000 results and you click to page 7 or 8 and it only has like 400...
Blogsearch only works well for Blogger - in terms of catching new content fast. I have a niche blogspot Blog with PR3 and it catches new posts in less than one hou, posting them in the main Google as well.
Why do I keep coming back to this post obsessively?
Well, it's been informative (grease monkey fools me sometimes too, but oh how I miss it when I'm use Internet Exploder) but I think it's really more about how fascinated I am by how many other people are frustrated with Google.
Search works great. Ad stuff works great.
The stuff that is easy (gmail, reader) works great.
The rest of it? Not so much.
It's like the old days when we got MS stuff shoved down our throats at work when Lotus, or Wordstar, or Quarterdeck (yes, I am old) worked so much better.
-OT
Temper? Clearly you've never worked with some of the people I've worked with!
I am often frustrated with Google's results, not the least because changing my C-block by going through a server in another hosting site will change which part the google index I'm seeing.
Swear to god once the one I get via RoadRunner at my house did not change at all one weekend. I was banging on a blog post, linking to it, trying to watch how Google search returned different results - Nothing.
Then I went through a webserver out in CA and went to google - there were all sorts of different responses.
So I got drunk instead.
-OT
As Joost commented, I also don't know why google is showing no-follow links in google webmaster tools. That makes me think that Google still might hold some value for no-follow links. I posted same query on other post about no-follow concepts, but didn't got any satisfactory answer.
I can see Yahoo Answers pointing to my site, and it is picked up by google webmasters tools even if they are no-follow links. If Google don't consider no-follow of any value why they still log it in webmasters tools.
Any idea why?
sahota, we wanted to give a pretty complete picture in our backlink display, so even though those nofollow links don't count, we do show them.
Would be nice to have an option to filter them though *nudge* *nudge* *wink* *wink*
:-)
Matt, I don't suppose you'd be kind enough to shed some light on this issue I blogged about a while ago? Even if it's just pointing me in the direction of some other articles writing about how Google copes with the difference and how you guys attempt to ensure relvancy with these kind of phrases? It's something I haven't seen much talk about and I think there's still a fair amount of work to be done in this area. Would love your thoughts!
Ah yes, IT. Part of the issue is that "it" can be a stopword in some cases, and upper/lowercase makes a difference, e.g. IT usually means something different than it.
There are very very few words where capitalizing makes a big difference in the word sense. One other example I can think of is polish vs. Polish. So even though if you go back and read the original Google pages, they talk about indexing the case of a document, in terms of effort vs. payoff the win is very low.
That said, I have heard people ask about this before. I'll send a pointer to your post to some webranking people, but I'll mention that in the past we've viewed as not as big a help as other things we could do.
Thanks for the reply Matt - are you saying that the capitalisation actually plays a part in determining which search results to show?
When I wrote that post (which was admitedly a while ago) I tried using both "IT" and "it" in search strings and couldn't see any difference in the rankings...
I agree though that simply looking at the capitalisation doesn't help that much since people will still search for "it support" when they mean "IT support" or "polish" when they mean "Polish". As you well know searchers are lazy after all!
If you could get someone to stop by and talk about it some more that would be fantastic! :-)
I obviously can't speak for Matt, but I think he is talking more about paying attention to case when spidering (because most people will use the correct capitalisation on their websites) in order to improve search results when they determine (through context etc.) that you mean IT rather than it (because you search 'it support, for example).
That feels like a convoluted explanation. Hopefully you get what I mean.
Tom_C, I don't know if they'll leave a comment, but they're already internally discussing why one improved way of handling the query didn't trigger.
Will - yeah re-reading I see what you're saying and that makes much more sense than taking the capitalisation out of the search string!
Matt - no problems, I'd love to be part of the discussion but then I guess that's what I get for persuing a career in SEO and not being a technical SERP engineer at Google ;-)
Thanks for taking a look and adding your input - very much appreciated!
While we're on the subject (of capitalisation, not you leaving to work at Google!), here's my latest post about the differences in handling highlighting between natural and paid results on the distilled blog.
Matt,
Like Tom, I would also like a way to determine which of these links is a nofollow or has been 301'd, even for the purpose of internal linking.
Please and Thank You.
Then tell us which ones are nofollowed and which aren't. You aren't giving away any PageRank info. And it appears that Joost has a script to tell the world that anyway. We have tens of thousands of links to the OneCall.com site and I'd like to know which are nofollowed and which aren't. Furthermore, tell me what the inbound link text is for the link. This will make it easier for me to target sites that are using silly anchor text to link to our site. Case in point was BigPictureBigSound.com using the term "Get a Deal, not the Shaft" to link to OneCall.com. Not at all a marketing phrase we want associated with OneCall. I only found it because he happen to be a unique phrase when I Googled it I found it easily. Only found the phrase because it was in the top 10 inbound phrases to our site.
Google needs to associate the link info and the anchor text info together so webmasters and SEOs can quickly find the sites that should improve their anchor text. I'd love to be able to call or email the sites that are linking to us with 'Click Here' (#11 inbound), 'Buy Now' (#10 inbound), 'Go to Store' (#5 inbound), 'Here' (#21 inbound), 'This Link' (#25 inbound), you get the point.
To complete my rant, I will point to Vanns.com. A competitor of OneCall's that I have submitted spam notices on, talked to Googlebods about at SES San Jose, etc.
https://www.vanns.com/shop/servlet/item/features/487019177 This page . . . come on, can you stuff any more keywords on the page? The ultimate overkill is the changing of the alt text on their own damn logo! Yet, nothing gets done and they continue to be one or two slots above me on most searches. FRUSTRATING!
Okay, ending rant . . . ;-) Thanks Rand, that felt good. ;-)
Thanks for respondiong Matt!
I know I am just some little dude, But it means alot to see guys like you getting invloved here:)
Plus Rand is soon to be the "world renoun seo" or the next google search seo :)
Is it me or is Google getting more and more "manual"? I mean they seem to be adjusting the results by hand, ranking who they choose. Even if they are using the algorythm, it can too be programmed to filter out specific sites.
I wonder why I don't hear much of that in Yahoo and Live. They don't seem to be picking on specific sites.
I don't know. Sorry for my ramblings, just thought I'd share.
"A very popular post I wrote a few weeks back called "Google PageRank Losses for Hundreds of Websites" still hasn't been indexed. Yahoo! has it. Live has it. Google's visited it with their spider plenty... What gives? There's almost 1000 links pointing to it."
That's evil.
I don't want to be presumptive here, but judging from the time that Mr. Cutts entered this thread (just below), I'm almost vain enough to think that perhaps he was the one that gave me the thumbs down.
I don't want to be presumptive, but if that is in fact the case - "Wow!" I'm honored. I'll take a thumbs down from Matt Cutts any day of the week. BTW Matt - great commentary below. Definitely not evil!
Sean, you got thumbed down by Dr. Evil..:)
Or how he calls it, a Wag of The Finger...
I don't want to be presumptive here, but judging from the time that Mr. Cutts entered this thread (just below), I'm almost vain enough to think that perhaps he was the one that gave me the thumbs down.
I don't want to be presumptive, but if that is in fact the case - "Wow!" I'm honored. I'll take a thumbs down from Matt Cutts any day of the week. BTW Matt - great commentary below. Definitely not evil!
It is very admirable that Matt Cutts takes a minute to clear some issues in this post. Many people seems to hate to Google now due to the last changes but, with a lot spam and paid links in anywhere on Internet, I guess is very difficult for Google to make tuning to its algorithm constantly. Anyway, Googe seems sometimes a little tyrant (maybe like Microsoft)
I agree , It was really nice of Matt,
I thjnk if more people followed the seomoz teams suggestions the whole world of search would be a better place
I was going make another puppet video for next week's PubCon show where Matt Cutts would play Santa and tell who has been naughty and nice this year.
But since Matt said,
I guesss I will not bother. Sounds like the SEO community is getting weaker in terms of link juice.
I thought the puppet video I did of you Matt for the SES conference on paid links would have gotten a link from your site but I guess it was not good enough or my site was not clean enough to link to.
It's not so much that the search community is getting weaker, the whole web is just getting bigger, and as a result of that, our community is becoming a smaller part of that whole web. To keep the same amount of PageRank flowing, our community would need to grow AT LEAST as fast as the whole web...
Wow. Some really juicy stuff going on today! No time to read the entire post and comments but you can bet your a** I'll be reading word-for-word later.
Odd that a partial logo from the BOTW Blog is displayed in there.
Is that a Webmaster Central thing or caused by Joost's script as well?
D'oh! Now that the screenshot was removed, this comment has no context...
I think that yes - that was part of Joost's script :)
your blog not being indexed is what I find most confusing/disturbing. It makes no sense at all. I like to drift into conspiracy mode and think there's human intervention but even that makes no sense because one would think that the G crew have better things to do.
I'll probably take a little slack for this comment but that is ok…
While I understand your complaints and have plenty of complaints of my own about Google related issues, I felt this post was very much like a public temper tantrum.
I'm glad to see posts questioning Google from Rand. 1, it keeps people believing that he is still one of the guys (not that anyone thought otherwise) and 2, it shows that everyone can have issues despite following best practices and doing whatever they are supposed to do.
It does seem like censorship on Google's part, although I doubt that is the case. It is strange that for a case study I bought a new domain, put together a new site in five minutes using very, very basic SEO principles, gave it one inbound link, and it was indexed and showing up in Google SERP's within three days. Rand's post is obviously far more worthwhile and nada.
Hello everyone, I wanted to publicly apologize to Rand for my earlier post where I chastised him for this post. I’ve been watching this post all day long and as NickNick said, a lot of great information has come from the comments of this post.
Maybe we should all complain a little bit more often about wrongs we feel about our websites or our client websites? No, I doubt Matt would enjoy spending his evening listening to all of us gripe.
Anyways, Rand if I’ve stepped on any toes I do hope you’ll forgive me and accept my apology for accusing you of what I accused you of earlier.
John Jones