As you start tracking your rankings and taking SEO more seriously, you're bound to ask the question (and we hear it a lot) – "What are ALL of the keywords that my site ranks for?" Sounds simple enough, but it turns out this question isn't just complicated – it's probably unanswerable.
I'm going to walk you through why it's such a tough question, discussing two myths that lead us to ask it in the first place. Then, I'm going to try to at least give you a partial answer – maybe not all, but enough to keep you busy for a long time.
Myth #1 – The Ranking Table
If you have any experience with programming, databases, or even just Excel, it's pretty easy to envision Google as some kind of giant ranking table. It might look something like this:
Keyword | Site | Rank |
---|---|---|
unicorns | www.unicorns.com | #1 |
unicorns | www.bobsunicornshack.com | #2 |
unicorns | www.unicornwrestling.org | #3 |
pretty unicorns | www.ohsoprettyunicorns.com | #1 |
pretty unicorns | www.sexymalibuunicorns.com | #2 |
pretty pretty unicorns | www.examplesgonetoofar.com | #1 |
While this approach might work for a very basic, closed system (like an internal knowledge base), it's not remotely practical on the scale of something like Google. The sheer scope of data, the blinding speed it gets updated, and the way that data has to be distributed across server farms (made up of thousands of servers), means that modern search is essentially a real-time calculation. There is no master table.
Myth #2 – Google Won't Tell Us
Ok, so it's not a table, but Google still knows what we rank for or they could figure it out, right? While Google definitely has plenty of data they won't let us see, some things are mysteries even to them. Back in 2007, Google's VP of Engineer, Udi Manber, shocked the search community by suggesting that as many as 20-25% of all Google queries were queries they had never seen before. Let's say that again – as many as 1/4 of all Google searches are new. Google later clarified that this is within a time window (not all of search history), but the number is still staggeringly high.
Much of this has to do with the fact that queries are naturally getting longer and more specific, with over half of search queries in 2010 being 4 words or longer. As people get more comfortable with asking detailed, natural-language questions, this trend is only going to continue. One way or another, your site is ranking for new keywords every day, and some of them are a surprise even to Google.
Tactic #1 – Mine Your Analytics
So, is figuring out what you rank for as elusive as the unicorns in my table? Fortunately, no. While you'll never know ALL of the keywords you rank for, you can definitely find a solid pile of data. Your best, first destination is your own analytics – here's an example from Google Analytics (go to "Traffic Sources" > "Keywords > "Non-paid"):
Of course, these are only keywords that drove clicks, but for my own site this represents 1,435 keyword phrases in just 1 month. My blog is hardly exceptional – it gets just over 200 visitors per day. So before you dismiss your analytics because they don't show you EVERYTHING, ask yourself if you've even come close to using the data they do provide.
Tactic #2 – Review GWT Keywords
The second place to look for keywords you're ranking for is Google Webmaster Tools, which is one of the only places to see data for keywords that drive search impressions but NOT clicks. Within GWT, go to "Your site on the web" > "Search queries", and you'll see something like this:
The "Clicks" column actually only goes down to "<10", so it's difficult to tell exactly which keywords drove no clicks, but comparing this data to your analytics data can help fill in some of the holes, if you really want to see the big picture.
Tactic #3 – Analyze Inbound Anchor Text
So, what if you want to find keywords that people aren't currently searching for but for which you could potentially rank? One place you might look is the anchor text that external sites use to link to your site, especially the longer tail phrases. For example, in our own Open Site Explorer, click on the "Anchor Text Distribution" tab and you'll get a full list of the phrases or terms external sites use to link to you (export to excel for up to 10,000 results):
For example, I would rank #1 for "muppet intern yoozer", if anyone actually ever typed that phrase (before I did today). I'm not sure how that helps me, but at least conceptually, seeing what phrases people are using to link to you can give you a sense of what you have the capacity to rank for, even if those phrases don't currently drive searches.
Stop Obsessing & Get to Work
So, maybe you can't find ALL the keywords you'd ever rank for, but so what? Using these techniques and extrapolating a bit (put in some quality time with Excel), you can easily generate a list of hundreds or thousands of keywords that you either currently or could potentially rank for. That ought to keep you busy for a while.
Why we are asking the wrong question here? The question should be 'What money keywords Do i rank for?'
My website ranks for hundred and thousands of keyword variations. Should i be bothered? If those keywords are not helping in generating any conversions/revenue/transaction then the answer is NO. I should be bothered about those keywords which are helping me in improving my bottomline in someway and which have the potential to generate more revenue/conversions if i rank better for them. Checking the ranks of all the keywords I rank for is neither practical nor a wise decision.
Yes! I rarely agree with you (no offense) but this is one time I can say you're absolutely right.
There is a major difference between keywords that convert and keywords that are used for just information purposes.
Couldn't have said it better myself.
LOL Jordan! ! I rarely agree with you (no offense) but this is one time It's a good thing Himanshiu has such thick skin!
I am glad you agree with me at least once :)
I had similar thoughts along the same lines Himanshu. Well said.
Nice article, but it begs the larger question, what specific, actionable steps should we take with that data?
If we're already ranking well for those key terms, it may make more sense to optimize for other terms. Your thoughts?
Sorry, I didn't intend this as a keyword research post, so much as an object lesson in why chasing after EVERY keyword you rank for isn't necessarily a good idea (or even possible). I'm a big fan of prioritizing SEO efforts. There are some great posts here on the blog about categorizing and sorting keywords:
https://www.seomoz.org/blog/keyword-research-using-categories
https://www.seomoz.org/blog/categorized-keyword-research-step-2-of-the-8step-seo-research-strategy
I agree with the idea that chasing after every keyword you rank for is often a fruitless effort. I think looking at GA and GWT should provide you with enough actionable data in the vast majority of situations. If you happen to need more, your idea of analyzing anchor text is useful, as would be looking at the keywords you target in html tags on your site, or applying topic modeling algorithms to your content which output the set of keywords that are the most topically relevant.
I really like the weighted sort feature in GA. I will sort by keywords and % new visits to find new keywords that are bringing new visitors. I find that new ranking keywords may be keywords just getting on the first page, or that are climbing up the page. You can put some effort into these new keywords and increase conversion opportunities and traffic.
Thanks Dr. Pete.
Dear Dr. Pete,
I love data, but my mommy told me that some people are mean and hate data. Why do people hate data? That makes me sad.
RyanOD
Seattle, WA
--
Seriously though, I do this type of research in conjunction with our paid search team. This has two benefits.
While I realize not everyone is running PPC campaigns, if you are and you aren't mining their data, you are missing out on a lot of great information.
Great post! And...thanks for tipping us off to Yoozer. Loved it.
"the fact that queries are naturally getting longer and more specific, with over half of search queries in 2010 being 4 words or longer."
wow thats interesting........make me pay attention more to longtail
There's a lot of potential there with Google Instant as well (even though I hate it). It's possible to target 'incomplete' words that aren't (yet) competitive. - Jenni
Google Instant can be a potential goldmine for creative conversion optimizers.
I know I usually type in full questions first when searching for help on a topic. I don't start diverting from longtail until I don't find any relative information. The average internet user would be using longtail searches for every search. Most people online are technically suave.
In myth 2 you mention the increase in the number of new searches Google see's. I don't know for sure, but I'd guess this is now on the wane. This is down to Google suggest pointing users down a handful of very specific paths and not allowing them to phrase their query the way they would want, were they forced to complete the query naturally.
I'm not saying this is a bad thing - it's certainly very handy for users and I use it all the time, but I would guess that it would impact the number of unique and new queries Google sees. Ranking for relevant Google suggest terms in your niche is a really important consideration these days and is one of the key things I look at when conducting keyword research.
I haven't seen data from the last year or so, but I have heard that Google Suggest is driving a sizable percentage of searches, and I suspect you're right. If people are using Suggest, it's going to decrease query diversity and may drive people to a mid-length search (2-4 words, probably). I have my doubts about whether that's a good thing.
Very Good Point. Add to this the rollout of Instant for Mobile and short tail still rules, however there is nothing wrong with extra gravy.
I would actually bet it has either increased or stayed the same; a big contributing factor for this is events and breaking news. Imagine all the new searches for the recent stories on Egypt, Libya and Charlie Sheen. This doesn't even get into niche stuff or communities.
I was very curious when I read the title of your blog post which tools or methods you would suggest but now I am satisfied.
We work mostly with Analytics data, GWT and OSE. As you wrote, "That ought to keep you busy for a while." It really does ...
Nite petra, how are you? i send you a private message!!! ;)
This reminds me of the first "SEO company" I worked for and the tactic they used to sell new clients and convince the existing ones they were getting stellar results.
They guaranteed "100 Page One Rankings."
My job was to mine the analytics and other sources and drop the keywords into rank checking software (IBP). It was a sad, sad task.
Of course, without fail, we'd end up with 50-75 new "page one rankings" for any given client. Classic example of an empty metric. Traffic/engagement can stay steady or even drop, you'll keep finding new page one rankings.
I wrote a YouMoz post detailing the entire process, "Guaranteed Rankings: Methodology of a Shady Business Tactic" - if anyone cares to look, it's rather amusing until you consider small businesses were getting burned in the process.
Yeah, sadly, I've heard that horror story from the business side, too. It's not that exciting to rank for "Justin Bieber Pre-concert Pancake Breakfast in Tuscaloosa".
I worked for an SEO company that used similar tactics. They never did anything as large scale of "100 Page One Ranking" but they would find terms a site is already ranking well for, which have next to no competition, and then develop a handful of inbound links to boost them to the top of the SERPs. Analytics mining paired up with Google Keyword tool was used to find the words that required the least amount of efforts, even if the search volume and competition on those search queries were slim to none.
Good summary - "Stop obsessing and get to work."
sites like semrush.com tells us what keywords we are ranking on. i think with the help of tools you mentioned, we can focus on keywords that helped us generate more clicks than just ranks.
Not to knock any particular tool, and I think SEMRush, Spyfu, etc. can be great for competitive research (especially on the PPC side), but I've found that they have massive gaps on the organic side. Typically, they only show a fraction of what you're ranking for. You often need to put multiple resources into play.
Dr. Pete, You hit the nail on the head. I don't think there is one tool that can give you the complete picture. Even if there was it is always smart to have diversity in your analysis tools.
Agreed, I never would use just one 'count' for the keyword indicators, but at least 3 different reports to see patterns in the keyword phrases. And also, Dr. Pete good call about anchor text.
Thx Dr. Pete - I have to talk Sr. Mgt down from the ledge all the time on this one... PPC search volumes in my opinion are still very inflated when compared to organic volumes. I've used both of the aforementioned tools and they're far more useful for evaluating PPC campaigns and early stage competitive research; there are gaps here too, especially with data refresh rate. You may be looking at data that is more than a month old in some cases.
What is your opinion using regex to pull out SERP # for key words from GA? I've had some success with this method but haven't nailed the magic formula yet (as it were). Obviously this will only give the SERP number for that key word and not actual rank but it strikes me as a fantastic starting point for developing a list of KW's to track. Any thoughts on this or resources you could point to?
SEMrush is an awesome tool for researching competitors FOR seo.
Here are some of SEOmoz's positions (reported) (that would be good to know if you were a similar company):
search engine optimization, seo blog, seo tools, search engine ranking, seo guide, search engine rankings, social media marketing, small business seo, seo tool, seo strategy, seo blogs, seo web development, search engine rank, seo strategy, ppc strategies, search engine positioning, search engine optimizing, seo keyword research, free seo tools, seo search engine optimization, popular searches, increase traffic, search engine optimizer, seo firms, seo rankings, seo tracking, website search engine optimization, cost of seo, guide to seo, how to increase blog traffic, seo for beginners, and last but not least.. rand fishkin. "seo software" is not on the list yet... but soon.
But as far as finding everything, no way. I would be interested to know out of their top "non-branded" search traffic to the site, how many of these words are in their top 25 keys.
I have a client that still wants to see a monthly keyword position report for their site. I've tried to explain to them that by the time the report is finalized and sent over to them, the data might be completely wrong! I agree that it's best to use your own analytics to see how well your keywords are working for you. Don't get so focused on the position, stick to traffic and conversion.
same here... not sure the best approach to explaining it to clients
While Google definitely has plenty of data they won't let us see, some things are mysteries even to them. Back in 2007, Google's VP of Engineer, Udi Manber, shocked the search community by suggesting that as many as 20-25% of all Google queries were queries they had never seen before
A similar stat just came from the SMX tweet stream either today or yesterday. It's actually quite amazing when you think about what that means.
Anyhow Pete, really nice post and I'm glad to see that you got a nice slot like not behind Oli'smassive noob infographic post. . :) I think you owe Jen a jar of Nutella for such perfect placement this time.
While I 'm sure I don't see nearlythe number of clients y'all do, I can say that in my limited experience I've never been asked "what keywords do I rank for?" THe much more common question is how can I get more business from my website?"
I think that if someone did ask me that question, I would answer them with another question or two. "What keywords do you want to rank for?" then follow up with a "Why?" to try to figure out the direction they should be headed.
Nice Article! I must say that through this activity to make myself busy for several continuous hours I can come up with list of hundreds of keywords that I am ranking for or I can potentially rank for… Obviously not every search term will be driving traffic to my website but further I can exclude the list of keywords that contain search volume and on which I can possibly rank for, in order to increase the search engine traffic and conversions.
The tools you mention specially ‘Open Site Explorer’ can be a wonderful choice to extract data for the website.
Thanks for sharing sir!
Hey Pete - great post!
Just a little reminder - our keyword tool runs rank on the first 10,000 keywords in your Google analytics (via an optional GA connection) account and fetches adwords API search volume for each keyword on exact match. You can categorise, group and filter keywords too.
https://tools.seogadget.co.uk/about
All the best,
Richard
I use the SEOGadget keyword tool for several websites. It pays for itself pretty much every month to be honest!
Initially it allowed us to easily see where we needed to conentrate to push a few key terms to get first page rankings. It now allows us to keep an eye on these rankings and helps us develop new tems and phrases to look at as well as an idea of the kind of traffic levels associated with the new terms.
Give it a go!
Hehe! Thanks curiousSEO - doing a little happy dance :-)
Well Mr. Baxter, I bought AWR by Caphyon a year or so back... because you told me to on the SEOmoz DVD's ;p
I might have to try your one out now then.
Excellent - I've been meaning to check the new keyword tool out, and that does sound like a great feature.
Thank you answered some questions that I had.
It is going to very hard to get keyword tracking 100% right, I think the best way as you said is to just get a massive list from all your sources i.e Google webamster tools data, Competitive data, traffic keyword reports and link data for anchor texts. I mean sure enough you may end up with around 1000+ keywords but just track every thing and then from the data in excel compile all the keywords you do rank for.
You can never get it right but people are always going to search for soem type of random keyword for example adding in numbers or what ever and a brand name to your website.
As said above SEMrush is ok for research and ideas but it is no way an accurate tool for keyword research, even Google can get it wrong alot of the time with some crazy data you see in webmaster tools.
I like that you point out checking analytics. Sometimes the best guide to the future is in the rearview. I can't tell you how often I pour over analytics data fishing for little niches of keywords to improve upon, only to find we're ranking for things I didn't expect. Great post.
Awesome post. Really enjoyed reading it and it really helps having all these insights in one place. One more option I would like to suggest is using the SEO keyword rank filter in Google Analytics mentioned the yoast post. This will add the ranks to the organic search keywords in Google Analytics and you can then filter it by using (position:1) and so on to identify all the keywords you are ranking for on each position.
https://yoast.com/track-seo-rankings-google-analytics/
Great Post. I checked my analytics account and I was surprised to see almost half of my traffic coming from key word phrases I would never have thought I would show up for. Very helpful!
set of tricks to use for a general keyword that might be in use worldwide, something like webhosting, this is generally used and wonder if a site like www.sailerhost.com can even come up at all.
Yeah. I agree we no need to bother for correct keywords. we need to use on these tools for SEO Services only when your keyword dont have traffic volume which would detractyour result on SERP.
I like using Tactic #1 to get a baseline of all the different terms people are using to find my site (sometimes, I'm surprised by the terms people use). I use that information to find good long-tail keyword opportunities.
By the way, Google Analytics only allows the export to CSV of 500 lines, right? Anybody know a way to get around that and download the entire list of all keywords driving traffic to the site?
Actually, Benjamin, there is a way to get more data out of GA! You have to append the URL to do this. While viewing this keyword report, look near the end of the URL. It will say Count and then list the number 500. Change 500 to the number of keywords you want to view and refresh. Depending on how large the number it is, it can take some time for GA to gather all the data. But then you can export away! I love this hack and use it for a lot of other reports I pull.
This article is old and things have changed so much. Any chance of revisiting the subject in 2016?
Why not use a tool like SEMrush to get this info quickly?
Maybe I'm missing the whole point here, but isn't that what Google analytics does? I visit Google analytics every day, it tells me how many people have visited my site, and also what key words/phrases have been used. How accurate the hit count is I couldn't answer, but the keywords seem to be genuine, all of them seem to relate directly to the content of my website. As for making money, if the keywords are relevant to your content and making money shouldn't be too difficult. My website is very small compared to a lot, probably the more than 30 pages, plus a community forum and I don't do too badly with Google AdSense each month.
This is a little more complicated than it first seems, and especially now that (not provided) is in place, which wasn't a factor when Dr. Pete wrote this.
One element is that you can rank for something, but if nobody has ever searched for that phrase, you'll have no way of knowing (via Analytics or Webmaster Tools) that you rank for that word or phrase. For example, I'm sure Moz now ranks for "seem to be genuine all of them seem to relate", yet it's unlikely anyone would search that.
This post is now a bit outdated, since Google does not send along the vast majority of the keywords people use for organic searches, so you can't just look in your analytics program and know what keywords people are using to find you. A peek at the blog stats for any recent post (in the past several months) will show just how much of our traffic from google is (not provided).
Hope this helps!
Dr. Pete, thanks for another great post. Great insight. Overall what i understand its like set theory. We need to collect different set of information and then find a correlation among those. Correct me if I am wrong.
These are all really great tactics to uncover keywords. Plus, with the more recent integration of Google Webmaster Tools in Google Analytics, you can accomplish Tactic #1 and #2 all in GA now. Like you said, there's certainly enough data to keep you busy for a while!
Want a super easy way to do this?
If you guys wanted to check your rankings, you could simply use Bright Local. I learned It from Daniel As My Network's Daniel Brown (tee hee). Anyways you can track lots of keywords and you can also give your clients an almost automatic update of their ranking with a press of a button. Literally... Not only that but you can also brand the emails you send out. If you wanted to check your own rankings it's works much in the same way and you can see it in real time. I love this tool and hope this helps you guys.
https://www.brightlocal.com/
Warm Regards
and remember
Get Electric (buzz buzz)
I can see the importance of knowing the more important key words and phrases that you rank for, but I am having trouble understanding why drilling down to all of these phrases and spending a great deal of time with this is going to benefit your optimization efforts if no one is looking for these terms on a regular basis.
I guess if you are in a very competitive niche this may be important?
Love the last section........Good article
There is no real value in knowing all the keyword phrases your rank for. It is an obsession with useless information. The only terms that really should matter are the ones that bring you qualified traffic. Spend the time doing your homework and creating a strategy that will bring you the qualified traffic you desire. "Stop obsessing & get back to work" is certainly the advice to take away and follow from this article.
With the use of proper tools (Google Analytics and Webmaster Tools) you can definitely harvest a good amount of ranking data, which you can then run with over to any type of "Rank Checking" tool (Advanced Web Report, SEOBook Toolbar, and some web-based rank checkers.)
This is how I handle a lot of my smaller sites, because the effort required is next to none, and the data you can extract is valuable for developing or tweaking an SEO campaign.
Agree: there is no value in measuring keywords that produce no traffic.
Is there a tool out there that can determine every single ranking a website has without you typing in the keywords?
I have been looking for ages.
Another note, i find that analtytics does a good job providing traffic via keyword placements. It's a very easy way to show clients that what you're doing is working. Especially if you can prove conversions through search keywords that aren't their brand. Which is what I am known for.
Great piece and couldn't agree more on the unplumbed depths of Goog's tools. In fact, ranked them #1 in a post I did on Top 10 tools. But loving and learning about OpenSite as well so it made my list too. Here's my list https://bit.ly/toolti2 Keep up the good work.
Google Analytics provide enough data about the rankings, also I used webmaster tools in accordance of analytics to find which is the major keyword that I ranked for & try optimizing that page for better ranking.
Open site explorer give some details about the backlinks for the website other then that I frequently use bluebacklink.com to find the usage of anchor text & nofollow tags, its a really handy tool.
The benefits of knowing your keywords can vary with what kind of site you're running, too. For an ecommerce site, you (should) pretty much know what kind of keywords you need to rank for. Now these tools can still do a handy job of helping you mine the specifics and know which ones are the most important or the most useful or help you consider angles you hadn't thought of, but you're less likely to stumble upon the unexpected in a way that's likely to affect your overall strategy. On the other hand, if, for instance, you have a blog that covers various topics and suddenly you find yourself ranking for a particular phrase that you weren't expecting, it might tell you, hey, people are interested in this topic, if this is something I am knowledgeable about maybe I should write more about it to drive more traffic to my blog.
Thanks for posting (have mostly been using GWT so far, and off course GA). Looking forward to try the Open Site Explorer, as soon as it gets some data on our site, https://adrf.co
Dr. Pete, thanks a lot for clearing that up. I've been getting questions like that from clients myself and my answer was very similar. Now I'm glad the community confirmed what I was thinking. Thanks again!
I always think it is funny when I see a chart from clients old SEO showing that the client ranks #1, #2, #3, etc. so many times and for random key words.
Client: "Look how awesome this is! Me: "How much traffic do you get from each key word?" Client: "???"
Anyway, Keyword research is a lot work and time in Excel!
I like the last point of your post about not obsessing and getting down to work. Keyword rankings are really just an ego stroke. What really matters is traffic that drives revenue. The question you should be asking is how can I increase revenue from organic traffic. There are two components to this: more traffic and higher conversion rate.
The third way( external website anchor text checking) is based on huge backlinks the website already get right...
Sorry, my sample table is misleading - you can actually run OSE out to up to 10,000 back-links, including keywords that are used by only one or a handful of sites. Across your full link profile, you're bound to pick up some interesting data.
I track long tail more then 1-2 word keywords and I do belive that long tail will be predominant in all searches very soon.
Really good post. As per your word this is the most tough question and your posting is a very good answer.
May I know what is your opinon in using keyword tools like spyfy, keywordspy? Because these keywords gives the ranking keywords of a website.
Hi Pete! Good information, thanks.
Besides the ones you told us, I've used sometimes: Awstats (from the Cpanel). Why? Because sometimes I've found there new keywords that nobody shows, I don't know why, but it happends, most of the times for spanish and italian keywords.
Cheers,Fran.
Why din't you talk about the Google Keywords tool anywhere is the post.
realy good article based on keywords and raking info.
i have a website www.hebrewpublishing.com
trying hard to higher page rank with all idea and instructions.
can you please help me achieve my goal to higher my page rank.
Pete, something you missing here is the next step to split the keywords and the keywords to improve because are most popular, After you collect the keyword from Analytics and Specially Webmaster Tools: You need to: Go to Google External Keyword Tool And adding 100 at the time search this keyword to find the most Popular Searches 2 Extra Settings to Apply to the external tool First be sure that you CHECK Only show ideas closely related to my search terms then do the search and Uncheck Broad and Check Exact Sort by the column Local Local Monthly Searches And Keep the keyword that get more than 100 Searches a Month. Repeat 100 more and Consume all the keywords you collect on the analytics and GWTools. After you filter all the batches, then is time to add the Most Populalr keywords and see what pages are ranking for this keywords, the do the on page work to make this pages more relevant and DONE a good SEO task. I do this Monthly for my Clients. and Works Perfect.