I'm currently working on re-authoring and re-building the Beginner's Guide to Search Engine Optimization, section by section. You can read more about this project here.

Where to Start with Web Analytics

That which can be measured can be improved, and in search engine optimization, measurement is critical to success. Professional SEOs track data about rankings, referrals, links and more to help analyze their campaigns and create road maps for success.

Recommended Metrics to Track

Although every business is unique and every website has different metrics that matter, the following list is nearly universal in appeal. Note that we're only covering those metrics critical to SEO - optimizing for the search engines - and as such, more general but still important metrics may not be included. For a more comprehensive look at web analytics overall, check out Choosing Web Analytics Key Performance Indicators from Avinash Kaushik's excellent Web Analytics Blog.

#1 - Search Engine Share of Referring Visits

Search Engine Traffic Share

Every month, it's critical to keep track of the contribution of each traffic source for your site. Broadly, these include:

  • Direct Navigation (type in traffic, bookmarks, email links without tracking codes, etc.)
  • Referral Traffic (from links across the web or in trackable email, promotion & branding campaign links)
  • Search Engines (queries that sent traffic from any major or minor web search engine)

Knowing the percentage and exact numbers will help you identify strengths and weaknesses and serve as a comparison over time for trend data. If, for example, you see that traffic has spiked dramatically but it comes from referral links with low relevance while search engine and direct type-ins fell, you'll know you're actually in much more trouble than the raw numbers would suggest. You should use this data to track your marketing efforts and to serve as a broad yardstick for your traffic acquisition efforts.

#2 - Visits Referred by Specific Search Engines

Search Engine Share

Three major engines make up 95%+ of all search traffic in the US (Yahoo!, MSN/Live & Google), and for most countries outside the US (with the notable exceptions of Russia, China, Japan, Korea & the Czech Republic) 80%+ of search traffic comes solely from Google. Measuring the contribution of your search traffic from each engine is critical for several reasons:

  • Compare Performance vs. Market Share - By tracking not only search engines broadly (as the screenshot above shows), but by country, you'll be able to see exactly the contribution level of each engine in accordance with its estimated market share. Keep in mind that in sectors like technology and Internet services, demand is likely to be higher on Google (given its younger, more tech-savvy demographic) than in arenas like cooking, sports or real estate (where the percentages might be closer to the estimates from firms like Comscore).
  • Get Visibility Into Potential Drops - If your search traffic should drop significantly at any point, knowing the relative and exact contributions from each engine will be essential to diagnosing the issue. If all the engines drop off equally, the problem is almost certainly one of accessibility. If Google drops while the others remain at previous levels, it's more likely to be a penalty or devaluation of your SEO efforts by that singular engine.
  • Uncover Strategic Value - It's very likely that some efforts you undertake in SEO will have greater positive results on some engines than others. For example, we frequently notice that on-page optimization tactics like better keyword inclusion and targeting has more benefit with Live & Yahoo! than Google, while gaining specific anchor text links from a large number of domains has a more positive impact on Google than the others.If you can identify the tactics that are having success with one engine (or that are failing to succeed with others), you'll better know how to focus your efforts.

If you find your site underperforming at one of the engines (based on broad market share numbers), don't immediately panic. Remember that search engines have demographics and biases just like any other referral source. For example, in the US, Google's market share is supposedly between 65-70%, yet the vast majority of sites we've ever worked with (and those reported by our friends and colleagues in the search marketing industry) show that 80-85% of traffic share from Google is actually far more common. A number of theories exist to support why this happens:

  1. Yahoo!'s top queries are navigational (their number one query is Google, for example), while Google's queries are more informational
  2. Many experts believe (and some have private data to suggest that) Yahoo! has a preference for sites participating in their paid inclusion program
  3. Yahoo! refers a large amount of traffic to Yahoo!'s own properties (Google, meanwhile, seems to have a similar love affair with Wikipedia)

Don't just rely on Comscore, Hitwise or Compete.com data to tell you what percentage of share an engine should provide - make sure to investigate. You can do this by running PPC ads on the various engines (and comparing impression data), checking rankings across the engines (if your Yahoo! rankings are just as good or better than your Google rankings, it's not missed opportunity, it's lower volume), and making sure you haven't made any dumb mistakes (blocking other engines' spiders, using the meta robots NOODP to control listings at Google, but forgetting to use NOYDIR at Yahoo!, etc.). 

#3 - Visits Referred by Specific Search Engine Terms/Phrases

Top Referring Phrases

The terms & phrases that send traffic are another important piece of your analytics pie. You'll want to keep track of these on a regular basis to help identify new trends in keyword demand, gauge your performance on key terms and find terms that are bringing significant traffic you're potentially under-serving (e.g., you rank well and get visits, but don't have content that helps the searcher accomplish their goal).

You may also find value in tracking search referral counts for terms outside the "top" terms/phrases - those that are important and valuable to your business. If the trend lines are pointing in the wrong direction, you know efforts need to be undertaken to course correct. Search traffic worldwide has consistently risen over the past 15 years, so a decline in quantity of referrals is troubling - check for seasonality issues (keywords that are only in demand certain times of the week/month/year) and rankings (have you dropped, or has search volume ebbed).

#4 - Conversion Rate by Search Query Term/Phrase

Conversion Rates of Various Search Queries

When it comes to the bottom line for your organization, few metrics matter as much as conversion. However, analytics often misstates the impact of conversion rates from the last referral, clouding the true picture of what brought a visitor who "converted." For example, in the graphic above, 4.46% of visitors who reached SEOmoz with the query "check backlinks" signed up to become members during that visit. What we don't know (at least, from this simple analysis), is how many of those visitors had already signed up, how many signed up during a later visit, or even what percentage of those visits were first-time visitors.

The real value from this sort of simplistic tracking comes from the "low-hanging fruit" - seeing terms/phrases that continually send visitors who convert and increasing focus on both rankings and traffic from that keyword referral as well as improving the landing pages that visitors reach. While conversion rate tracking from keyword phrase referrals is certainly important, it's never the whole story. Dig deeper and you can often uncover far more interesting and applicable data about how conversion starts and ends on your site.

#5 - Number of Pages Receiving at Least One Visit from Search Engines

Pages Getting Google Traffic

Knowing the number of pages that receive search engine traffic is an essential metric for monitoring overall SEO performance. From this number, we can get a glimpse into indexation (how many pages the engines are keeping in their indices from our site), and, more importantly, watch trends over time. For most large websites (50,000+ pages), mere inclusion is essential to earning traffic, and this metric delivers a trackable number that's indicative of success or failure. As you work on issues like site architecture, link acquisition, XML Sitemaps, uniqueness of content and meta data, etc. the trend line should rise, showing that more and more pages are earning their way into the engines' results. Pages receiving search traffic is, quite possibly, the best long tail metric around.

While other analytics data points are also of great importance, those mentioned above should be universally applied to get the maximum value from your SEO campaigns. Additional sources to read on this topic include:

Free Analytics Software

Many very high quality analytics products are available entirely for free. These can be installed either on your web server to collect and analyze log-file based data or in the code on your pages (as javascript) to capture individual visit data. Without software, you're up a creek - raw log file analysis is extremely tedious and time consuming and many organizations don't even have the ability to access their logs. Use software and track - and don't worry - the free options are not only better than nothing, they're pretty darn good.

Recommended free analytics software packages include:

While choosing can be tough, at the time of publication, our top recommendation is for Google Analytics (so long as you have few privacy concerns and don't mind the brief data delays), followed closely by Clicky. Once the Yahoo! Web Analytics beta opens to the public, that would also be a top suggestion (and SEOmoz itself has run on Indextools/Yahoo! for the last 3 years). If you cannot use tracking code on your web pages and need a log-file based solution, AWStats is our top recommendation, though any log file based tracking will suffer from the inability to track clickstream paths, first time vs. referring and other important metrics as accurately as cookie/session based software.

Paid Analytics Software

There are dozens (possibly hundreds) of paid analytics solutions, but for the purposes of this guide, we'll list only the most popular services:

Unfortunately, we don't have enough experience to recommend one particular package over the others, but you can read some very good analysis and comparisons, including:

Metrics for Measuring Search Engine Optimization

In organic SEO, it can be difficult to track the specific elements of the engines' algorithms effectively given that this data is not public, nor is it even well-researched. However, a combination of tactics have become best practices, and new data is constantly emerging to help track direct ranking elements and positive/negative ranking signals. The data points covered below are ones that we will occasionally use for our clients' campaigns and have proven to add value when used in concert with analytics.

Metrics Provided by Search Engines

We've already discussed many of the data points provided by services such as Google's Webmaster Tools, Yahoo! Site Explorer and Microsoft's Webmaster Tools (in part 8: Search Engine Tools & Services). In addition to these, the engines provide some insight through publicly available queries and competitive intelligence. Below is a list of queries/tools/metrics from the engines, along with their respective applications:

  • Google
    • Google Site Query - e.g., site:seomoz.org - useful to see the number and list of pages indexed on a particular domain. You can expand the value by adding additional query parameters. For example - site:seomoz.org/blog inurl:tools - will show only those pages in Google's index that are in the blog and contain the word "tools" in the URL.
    • Google Link Query  - e.g., link:www.seomoz.org - unfortunately, in 2004, Google removed most of the value from this query by changing the results to show only a sample (and not even a relative or consistent pattern sample) of links. These can include nofollowed links as well, and are not ordered by importance. We don't recommend employing this query.
    • Google Trends - available at Google.com/Trends - this shows keyword search volume/popularity data over time. If you're logged into your Google account, you can also get specific numbers on the charts, rather than just trend lines.
      _

      _
    • Google Trends for Websites - available at Trends.Google.com/websites - this shows traffic data for websites according to Google's data sources (toolbar, ISP data, analytics and others may be part of this). A logged in user account will show numbers in the chart to indicate estimated traffic levels.
      _

      _
    • Google Insights for Search - available at google.com/insights/search - this tool provides data about regional usage, popularity and related queries for keywords.
  • Yahoo! 
    • Yahoo! Site Query - e.g., site:seomoz.org - note that a standard site query will automatically redirect to Yahoo!'s Site Explorer, but advanced queries that include additional parameters such as site:seomoz.org inurl:rand will show Yahoo!'s standard results format. You can use these much in the same way as the Google site query to see the number and list of pages Yahoo! has in their index for a particular site.
    • Yahoo! Link & Linkdomain Queries - e.g., linkdomain:seomoz.org - as with site queries, these will redirect to Yahoo! Site Explorer unless additional parameters are employed. For example, to see only links to SEOmoz.org that have the word "google" in the title tag, you'd use the query - linkdomain:seomoz.org intitle:google. Yahoo!'s link queries are the most robust and accurate of the major engines, but do include nofollow links (and don't separately mark these, which can cause trouble separate value-passing links).
  • Microsoft
    • MSN Site Query - e.g., site:seomoz.org - just like Yahoo! & Google, MSN allows for queries to show the number and list of pages in their index from a given site. Unfortunately, MSN's counts are given to wild fluctuation and massive inaccuracy, often rendering the counts themselves useless.
    • MSN IP Query - e.g., ip:216.176.191.233 - this query will show pages that Microsoft's engine has found on the given IP address. This can be useful in identifying shared hosting and seeing what other sites are hosted on a given IP address.
    • MSN AdCenter Labs - available at adlab.microsoft.com/alltools.aspx - a great variety of keyword research and audience intelligence tools are provided by Microsoft, primarily for search and display advertising. This guide won't dive deep into the value of each individual tool, but they are worth investigating and many can be applied to SEO.
  • Ask.com
    • Ask Site Query - e.g., site:seomoz.org inurl:www - Ask.com is a bit picky in its requirements around use of the site query operator. To function properly, an additional query must be used (although generic queries such as the example above are useful to see what a broad "site" query would normally return).
  • Google Blog Search
    • BlogSearch Link Query - e.g., link:www.seomoz.org/blog - Although Google's normal web search link command is not particularly useful, their blog search link query shows generally high quality data and can be sorted by date range and relevance.

Employing these queries & tools effectively requires that you have an informational need with an actionable solution. The data itself isn't valuable unless you have a plan of what to change/build/do once you learn what you need to know (this holds true for competitive analysis as well).

For more detail, see the Professional's Guide to Advanced Search Operators, an extremely detailed and thorough resource on this subject.

Applying Data to Your Campaigns

Just knowing the numbers won't help unless you can effectively interpret and apply changes to course-correct. Below, we've taken a sample of some of the most common directional signals provided by tracking data points and how to respond with actions to improve or execute on opportunities.

Fluctuation in Search Engine Page & Link Count Numbers

The numbers reported in "site:" and "link:" queries are rarely precise, and thus we strongly recommend not getting too worried about fluctuations showing massive increases or decreases unless they are accompanied by traffic drops. For example, on any given day, Yahoo! reports between 800,000 and 2 million links to the SEOmoz.org domain. Obviously, we don't gain or lose hundreds of thousands of links each day, but the variability of Yahoo!'s indices means that these numbers reports provide little guidance about our actual link growth or shrinkage.

If you do see significant drops in links or pages indexed accompanied by similar traffic referral drops from the search engines, you may be experiencing a real loss of link juice (check to see if important links that were previously sending traffic/rankings boosts still exist) or a loss of indexation due to penalties, hacking, malware, etc. A thorough analysis using your own web analytics and Google's Webmaster Tools can help to identify potential problems.

Falling Search Traffic from a Single Engine

If a single engine is sending you considerably less traffic for a wide range of search queries, a small number of possibilities exist:

  • You're under a penalty at that engine for violating search quality or terms of service guidelines. Check out this post on how to identify/handle a search engine penalty.
  • You've accidentally blocked access to that search engine's crawler. Double-check your robots.txt file and meta robots tags and review the Webmaster Tools for that engine to see if any issues exist.
  • That engine has changed their ranking algorithm in a fashion that no longer favors your site. Most frequently, this happens because links pointing to your site have been devalued in some way, and is especially prevalent for sites that engage in manual link building campaigns of low-moderate quality links.

Identify the problem most likely to be the culprit and investigate. Forums like Cre8asit Forums, HighRankings and Google's Groups for Webmasters can help.

Falling Search Traffic from Multiple Engines

Chances are good that you've done something on your site to block crawlers or stop indexation. This could be something in the robots.txt or meta robots tags, a problem with hosting/uptime, a DNS resolution issue or a number of other technical breakdowns. Talk to your SysAdmin, developers and/or host and carefully review your Webmaster Tools accounts and analytics to help determine potential causes.

Individual Rankings Fluctuations

Gaining or losing rankings for a particular term/phrase or even several happens millions of times a day to millions of pages and is generally nothing to be concerned about. Ranking algorithms fluctuate, competitors gain and lose links (and on-page optimization tactics) and search engines even flux between indices (and may sometimes even make mistakes in their crawling, inclusion or ranking processes). When a dramatic rankings decrease occurs, you might want to carefully review on-page elements for any signs of over-optimization or violation of guidelines (cloaking, keyword stuffing, etc.) and check to see if links have recently been gained or lost. Note that with sudden spikes in rankings for new content, a temporary period of high visibility followed by a dramatic drop is common (in the SEO field, we refer to this as the "freshness boost").

Don't panic over small fluctuations. With large drops, be wary against making a judgment call until at least a few days have past. If you run a new site or are in the process of link acquisition and active marketing, these sudden spikes and drops are even more common, so simply be prepared and keep working.

Positive Increases in Link Metrics without Rankings Increases

Many site owners worry that when they've done some "classic" SEO - on-page optimization, link acquisition, etc. they can expect instant results. This, sadly, is not the case. Particularly for new site and pages and content that's competing in very difficult results, rankings take time and even earning lots of great links is not a sure recipe to instantly reach the top. Remember that the engines need to not only crawl all those pages where you've acquired links, but index and process them - given the almost certain use of delta indices by the engines to help with freshness, the metrics and rankings you're seeking may be days or even weeks behind the progress you've made.

And with that, ladies and gentlemen, the Beginner's Guide content is complete! Actually, I still need to write up the very important appendices, including the glossary, list of links to other resources, and credits, but I'm hopeful to get this done soon (and it's about time - I started way back in October of 2007!).

As always - comments, criticisms and recommendations are greatly appreciated.