I'm currently working on re-authoring and re-building the Beginner's Guide to Search Engine Optimization, section by section. You can read more about this project here.

Keyword research is one of the most important, valuable, and high return activities in the search marketing field. Through the detective work of puzzling out your market's keyword demand, you not only learn which terms and phrases to target with SEO, but also learn more about your customers as a whole. The usefulness of this intelligence cannot be overstated - with keyword research you can predict shifts in demand, respond to changing market conditions, and produce the products, services, and content that web searchers are already actively seeking. In the history of marketing, there has never been such a low barrier to entry in understanding the motivations of consumers in virtually every niche - not taking advantage is practically criminal.

How to Judge the Value of a Keyword

Every search phrase that's typed into an engine is recorded in one way or another, and keyword research tools like those described below allow us to retrieve this information. However, those tools cannot show us (directly) how valuable or important it might be to rank for and receive traffic from those searches. To understand the value of a keyword, we need to research further, make some hypotheses, test, and iterate - the classic web marketing formula.

The following is a basic, but valuable process for determining a keyword's value:

1. Ask yourself - is the keyword relevant to the content your website offers? Will searchers who find your site through this term find the likely answer to their implied question(s)? And will this traffic result in financial rewards (or other organizational goals) directly or indirectly? If the answer to all of these questions is a clear "Yes!," proceed...
2. Search for the term/phrase in the major engines - are there search advertisements running along the top and right-hand side of the organic results? Typically, many search ads means a high value keyword, and multiple search ads above the organic results often means a highly lucrative and directly conversion-prone keyword.
3. Buy a sample campaign for the keyword at Google AdWords and/or Yahoo! Search Marketing. Choose "exact match" and point the traffic to the most relevant page on your website. Measure the traffic to your site, and track impressions and conversion rate over the course of at least 2-300 clicks (this may take only a day or two with highly trafficked terms, or several weeks with keyword in lesser demand).
4. Using the data you've collected, make an educated guess about the value of a single visitor to your site with the given search term or phrase. For example, if, in the past 24 hours, your search ad has generated 5,000 impressions, of which 100 visitors have come to your site and 3 have converted for total profit (not revenue!) of $300, then a single visitor for that keyword is worth approx. $3 to your business. Those 5,000 impressions in 24 hours could probably generate a click-through rate of between 30-40% with a #1 ranking (see the leaked AOL data mining for more on potential click-through-rates), which would mean 1500-2000 visits per day, at $3 each, or ~$1.75 million dollars per year. No wonder businesses love search marketing!

Of course, even the best estimates of value fall flat against the hands-on process of optimizing and calculating ROI. Remember that the time and money you invest in a search marketing campaign must be weighed against any returns, and even though SEO is typically one of the highest return marketing investments, measuring success is still critical to the process.

Understanding the Long Tail of the Keyword Demand Curve

It's wonderful to deal with keywords that have 5,000 searches a day, or even 500 searches a day, but in reality, these "popular" search terms actually make up less than 30% of the overall searches performed on the web. The remaining 70% lie in what's commonly called the "long tail" of search. The tail contains hundreds of millions of unique searches that might be conducted a few times in any given day (or even only once, ever!), but, when taken together, they comprise the majority of the world's demand for information through search engines.

Understanding the search demand curve is critical, because it stresses the importance of "long-tail" targeted content - pages with information not directed at any particular single, popular query, but rather simply exposing the myriad of human thought, research, and opinion to the spiders of the search engines. As an example, below I've copied a chart of Moz's own keyword traffic, illustrating the small number of queries sending larger amounts of traffic alongside the plethora of rarely-searched terms and phrases that bring the bulk of our search referrals:

In the graph above (courtesy of Enquisite's amazing long tail KW referral tracking), you can see that the bright green terms in the head make up only a tiny fraction of the overall traffic, while the mid and dark blue areas comprise a considerable majority, and yet contain terms that have only brought 1-2 visits to the site. The numerical chart below shows this in greater detail - note that our top phrases bring in only a minuscule percent of our total search traffic.

Ignore the long tail at your peril - search marketing and web site content strategies must allow for this "impossible to predict" form of visits or risk losing out to a more expository and prolific competitor.

Keyword Research Sources

Where do we get all of this knowledge about keyword demand and keyword referrals? From research sources like these:

• Google Adwords' Keyword Estimator Tool
• Google Trends Keyword Demand Prediction Tool
• Microsoft AdCenter Keyword Forecasting Tool
• Wordtracker's Free Basic Keyword Demand Tool
• KeywordDiscovery Free Basic Keyword Demand Tool

Here's an example of mining keyword research data from Google's AdWords Estimator Tool:

We can see that Google is predicting both the cost for running campaigns for these terms as well as estimates of the number of clicks a campaign might receive. You can use these latter numbers (under the "estimated clicks/day" column) to get a rough idea of how popular a particular keyword or phrase is in comparison to another. The green, horizontal bar in the "search volume" column can also help to show comparative estimates of demand.

Other, less popular sources for keyword information exist, as do tools with more advanced data, and these are covered excellently in Rebecca's Professional's Guide to Keyword Research.

Keyword Difficulty

In order to know which keywords to target now (and which to pursue later), it's essential to not only understand the demand for a given term or phrase, but the work required to achieve those rankings. If mighty competitors block the top 10 results and you're just starting out on the web, the uphill battle for rankings can take months or years of effort bearing little to no fruit. This is why it's essential to understand keyword difficulty.

To get a rough idea of the level of competition faced for a particular term or phrase, the following metrics are all valuable:

• Search Demand Volume (how many people are searching for this keyword)
• Number of Advertising Competitors
• Strength (age, link power, targeting, and relevance) of the Top 10 Results
• Number of Search Results (it can be valuable to use advanced operators like "exact search" or the allintitle, allinurl operators here as well - see more on using these specialized searches for keyword research here)

Of course, if you'd like to save time, Moz's own Keyword Difficulty Tool does a good job collecting all of these metrics and providing a comparative score for any given search term or phrase.

As always, comments and constructive criticism are appreciated. You'll note that I'm trying to go back to making this more of a true "beginner's" guide, as I'm concerned that I've gone a bit too in-depth with previous sections. Hopefully once SMX West is over, I can devote more time to finishing this mammoth undertaking :)