October 18th, 2011, the day Google announced "Secure Search," was a dark day for many search marketers. We had hope, though; we were told only a small fraction of search referrals from Google would apply. This was proven false in just a few weeks as (not provided) quickly hit 10+% for many sites. Then, a year later, seemingly out of the blue, Google started to encrypt almost all searches. Today, we are approaching the dreaded extinction of Google organic keyword data:
Oh keywords, how I will miss thee.
Knowing the keywords that send us traffic from Google Search has always been a major pillar on which search marketers execute and measure the effectiveness of an SEO strategy. With Google "Secure Search" and keywords being stripped from the referral string, it's starting to look more like a crutch—or worse, a crutch that will very soon no longer exist at all. Here are five ideas and two bonus resources to help nurse keyword targeting and search ROI back to health. Will they solve all your problems? No. Will they inform a direction for future "provided" solutions? Maybe. Are they better than nothing? Most definitely.
1. Use custom variables to tag content with categories/topics
Most web analytics software allows site owners to pass custom variables through. In Google Analytics, a custom variable can be inserted into your code, and as the name implies, you can pass custom name/value pairs of your choice. It's one of the most useful analytics tools for web traffic segmentation with many different applications. Mix this functionality with category, topics or tags from a page on your site and you can now analyze your organic web traffic based on those variables. If you are discipline and creative in understanding and tagging your content, you will get insight about what topics are sending your traffic.
If you have some programming chops and can extract these variables from your CMS yourself and append them to your tracking code, more power to you! If not, and you are a WordPress user, I have some good news: There is a free plugin from our friends at Yoast. Install it and then simply select the following:
Once it is in GA there are several ways to get at the data. One is to simply go to Acquisition > Channels > Organic Search, then select the primary dimension of "landing page" and the secondary dimension with your custom variable. You now have a list of your landing pages that received organic traffic and the categories/tags related to each. Valuable stuff.
If you want some ideas of what tags you should be using, there are several auto-tag generator plugins for WordPress, Zemanta being one.
Requirements:
-
Programming chops or WordPress and Google Analytics
-
Being disciplined about entering tags and categories
Watch out:
-
It's human-powered, for better or for worse, and your data is only as good as the humanoid at the controls of your CMS
-
Doesn't help for long-tail targeting and reporting
2. Combining rank data with landing pages from Google Analytics
We can recapture some Google keywords by joining our rankings and analytics data. Download your rankings data from your favorite rankings tool; the more data you have the better. In Google Analytics, go to Channels > Organic Search > Source = Google and add the secondary dimension of "Landing Page." View the maximum number of rows and download the data into a CSV. Put your data in two separate tabs in a spreadsheet. Now, all you need to do it join the keywords from the rankings tab with the keywords from the analytics tab. This can be done using VLOOKUP. While you're at it, add the ranking data to the analytics tab. The end result will look like this:
Requirements:
-
Rankings data
-
Google Analytics data
-
Basic Excel or Google Spreadsheet skills
Watch out:
-
Using the method above with VLOOKUP will only return one keyword per landing page. With some crafty Excel work, you can figure out how to get all the keywords for that page
3. Site search: what users are searching for on your site
If you get enough people using the search feature of your site, it can be a gold mine for keyword data. After all, this keyword data will always be "provided." Configuring Google Analytics to capture your internal search traffic is pretty straightforward. Once you have done so, you will be able to see the top keywords people are searching for on your site.
Step 1: Open the Google Analytics profile you want to set up Site Search for
Step 2: Navigate to Admin > Settings and scroll to the bottom for "Site Search Settings." Enter in the parameter that is designated for a search query on your site; for example /search_results.php?q=keyword. If you use a POST-based method and do not pass through a parameter in the URL you can either configure your application to append one, or you can trigger a virtual pageview in your Google Analytics snippet, such as:
analytics.js: ga('send', 'pageview', '/search_results.php?q=keyword')
The category option allows you to look for an additional query parameter that can later be used to group the site search data. For example, if you had search on your site in different sections that you wanted to keep separate: help, content, documentation, etc.
Step 3: Let GA collect some data for a day or so and check out your results. Navigate to Behavior > Site Search > Search Terms to see a complete list that users search for on your site. To dig deep add the secondary dimension of "destination page" to see where the user landed after seeing the search results. Then, be sure to check out the secondary dimension of "search refinement" to see which keywords your users searched for after they searched for the original content. This can clue you into gap between what people are looking for and not finding on your site.
Requirements:
-
A search box on your site
-
Google Analytics
Watch out:
-
It's a limited data set (on Moz only about 1/2 or one percent of visits end up using our search)
4. Google (and Bing) Webmaster Tools
Google has created the headache with "Not Provided," but they have also given us a bit of medicine in the form Webmaster Tools. Released a few years back within Webmaster Tools, "Search Queries" provides webmasters with some basic information around their keywords, including average position, impressions, number of clicks, and click-through rate (CTR).
This data should be used, but has a few major limitations. First, only a small, Google-selected subset of the keywords is represented. There is no transparency about how or why they select the keywords, so using it to measure results of specific content optimization efforts can be inaccurate and even misleading.
Second, the data is limited to 90 days. If you ranked for a query 91 days ago, you'll never know. Webmaster Tools also has an API, but unfortunately the "search queries" data isn't available through it yet. According to Mr. Cutts, that is imminent. If you want to store your data for longer than 90 days and know how to program, you can use this PHP library or this Python library.
Finally, there is a limitation in how you can use Webmaster Tools data in Google Analytics. The good news is that you can integrate this data into Google Analytics with some basic authentication between the services. The bad news is that you can only segment the data in Google Analytics with 2 dimensions: country and Google property. Joining this data with behavior, demographics, goals, etc. would be extremely valuable.
Requirement:
-
Google Webmaster Tools account
Watch out:
- (Limitations noted above)
5. Deeper topical analysis
Avinash Kaushik, one of my favorite speakers MozCon this year wrote about understanding the "personality" of the page as a future solution for "not provided". He says:
"I wonder if someone can create a tool that will crawl our site and tell us what the personality of each page represents. Some of this is manifested today as keyword density analysis (which is value-deficient, especially because search engines got over "density" nine hundred years ago). By personality, I mean what does the page stand for, what is the adjacent cluster of meaning that is around the page's purpose? Based on the words used, what attitude does the page reflect, and based on how others are talking about this page, what other meaning is being implied on a page?"
I think this could be accomplished by performing topical analysis on body content of pages as they are published and then passed through to Google Analytics with custom variables; similar to what I described above with categories. This could be done by using DBpedia and one of the annotation open source application that uses it, such as DBpedia Spotlight. Spotlight detects mentions of terms in your content and scores the relevance of those mentions against structured data created from Wikipedia. Once the topics of the page are "extracted" and passed to your web analytics platform, you'll be able to use it as a dimension against organic search referrals to landing pages. (Thanks to Jay Leary for walking me through Spotlight)
Bonus: some other "not provided" resources
Mike King is not too worried about "Not Provided." His deck argues we should be focusing on segmenting our data by personas and affinity groups, and paying more attention to "implicit" rather than "explicit" intent. Good stuff.
Ten industry experts, including two Mozzers, weigh in here and answer a series of questions on the "Not Provided" landscape, including tools and techniques that they use, and even a few "Top Tips for 2014."
Conclusion
Keyword data from Google organic search is owned and controlled by Google and can never be replaced. Secure Search is here to stay and nearing 100%. There is no cure-all solution. That being said, search marketers are a GSD and generous group, and will continue to hack away at the problem and share solutions. What are some of the data sources and hacks you are using to deal with "not provided?" Are there future algorithmic solutions to this problem, or are we doomed to have to take our Google medicine and be happy with what they decide to provide in Webmaster Tools?
I have added already analytics to Webmaster . But was unaware of this.
Yeah Agree with Yogesh and Jinnat.
But working along with these data, is not enough to get rid over not provided data. You need to make analysis on GWT and Semrush data to run a effective SEO strategy.
GWT impression and clicks data really help to understand the organic keywords ranking and performance over SEs. But, how would you judge a keyword is driving good traffic?
My point is, earlier we were able to know a keyword performance like we had bounce rate, average time for every keyword which helped us to run page to page based analysis. Process was very short but now it's become long.
Now I take CTR and Impression data and get landing page data from Search Engines. Then I match with PageViews data (set with secondary dimension to source/medium) and find Exit Rate of every page which matches with queries and landing page.
This way looks more effective to me but it takes time and efforts. It seems Google want us to give more time and efforts now :)
Hey Sundar, very good point: with keywords no longer being provided by Google the time it takes to understand the keywords and/or topics that send traffic to your page can be prohibitive. I am hopeful that smart software folks will solve this combining available data, predictive algos and good UI.
Thanks Tim, Hope so....
Thanks for the insights. You definitely offered some good insights for combining the data. I have been definitely using webmaster tools to get an idea of what keyword phrases are driving traffic to websites. One other suggestion, however, I am not sure how accurate it can be, is using the organic keyword data feature in SEM Rush and it will give you some more keyword phrases that may be driving traffic to your website.
Thanks for some more tips, Tim. A couple are over my head, but with some practice and more research could be applicable. Your note, "This data should be used, but has a few major limitations." when talking about GWT Search Queries is spot on. It definitely has major limitations to interpret the data and the keywords are certainly not a full list.
Good one...webmaster is a good option to take insights... :)
Already added webmaster account to Google Analytics to get keywords(search engine queries). But I didn't know about this tactics. Let me try that. Good job MOZ!
I am still figuring out the way to get this figured out Ronnel. I will say thanks ones I see it actually working for me.
Thanks for sharing such an awesome post! Will work on it to conclude to the best solution for my clients.
Thanks for giving such nice tips. I was needed it as for most of the client, we are unable to check ranking keywords. Thanks a lot. I would definitely try it.
Sweta
Excellent post Tim! Right on the money. Its quite simple, USER EXPERIENCE! The main reason that they removed keyword data was to improve user experience. We as marketers were able to rank for major keywords and that was great except for the user that was using long tail keywords. Why would we focus on those long tail queries if they represented such a small percentage of overall traffic.Made us pay more attention to conversion rates also! Well this unfortunately didn't coincide with Google vision on handling longer queries so then the birth of (non-provided) stopped us in our tracks! Google simply wants us to be more rounded, more complete marketers so the un-experienced Googler can still find a Great site. So combine what this post taught you with covering more ground and at that point the traffic should be pouring in. Remember we are Inbound Marketers!
Good advice. While it sucks to not have the keyword data straight from Google, all is not lost. It's still possible to get good insight on your website's performance.
Really nice analysis here & please don't remind me about that day. Here is one another way to try out - https://www.niswey.com/blog/how-to-capture-not-provided-data/
Great advice and blog post as always from the Moz blog!
Great post some useful stuff here!!!
A handful of good tip to try and get keyword data. As ranking are becoming more and more personalized, the development of quality landing pages will become your topic traffic source. Thanks @Tim for sharing!!
Tim,
Great relevant piece here. Although I'm a little confused with Step #2.
Obviously you can make some educated guesses about what keywords you're performing for when looking at landing page data, but either it's still early in the morning or I don't get it…
Are you just saying to analyze the landing pages with the keywords you're ranking for to see what (likely) keywords are driving traffic to those pages? Just wondering if there's a specific analysis past that I missed or am unaware of.
Thanks for a nice piece that recapped one of the biggest events of 2013!
Hey Brady, yes that is what I am saying. Obviously the biggest issue is that you are limited to the number of keywords that you are tracking, and it doesn't tell you anything about the long tail. I'm sure Google has seen a big spike in bots scraping for rankings :)
Thanks Tim for sharing a such good post.
Client always pressurized that why we are not able to provide all info for the user " Keywords Not provided data" . It will help us more.
Is Google opening up “Not provided”?
I read this Blog: https://goo.gl/jHlqc2 a positive surprise came on 20th November, 2015, when Google started providing more detailed information about “not provided” keywords.
I would love to see sites have actual forum/thread ability on what would be considered their marketing pages. This would allow for real user interaction right at the place they are most interested. You could then data mine the specific keywords/questions/requests and be highly pro-active to your audience...
Also lets not for get the wealth of information page-level VOC surveys can provide. "Where you able to find what you were looking for? If not, why not?" We use these very effectively for certain clients that do not have direct connections with their clients to provide valuable feedback. Lots of great keywords.
SEMRush & AdWords Keywords Planner can give you insightful data.
Tim,
When I see an informed, reasonable post such as this on Not Provided, a huge smile washes across my face. Nicely done. The elements regarding deeper topical analysis are more nuanced than my own thoughts but are similar to something I have imagined as a way of better understanding a piece/page of content in relationship to the whole of the site, and discerning how and why prospects interact with the page, and how we can influence that interaction. Is it what they expected? What other pages reflect similarly? How similar pages/clusters of pages are "behaving." Avinash is always on-point.
RS
Hey Ronell, glad you enjoyed the post! Check out Mike King's thoughts on using segmentation and personas to bubble up insights through GA. I link to it above.
I see why you linked to it. Great insights, you guys rock!
Webmaster search queries can give you a great idea of what keywords are sending traffic to your site and internal site search data can give you the hit of what visitor is actually searching for within your website. These two methods are best one and also easy as compair to others to get keywords data.
My preferred method has been to analyse ranking and landing page metrics. It is also important to remember that Bing is your friend ;)
Yes, Bing is your friend! Thanks for pointing that out. Dr. Pete and Matt Peters, our resident data scientists, found single-digit percentage overlap with Bing keywords and Google. The long-tail is massively under-represented. But, still, we need to scratch and claw for all the data we can get.
Thanks for the tips, Tim. Especially #3 - I was not aware that we could do this. Very helpful!
Hi Tim,
Thanks for giving solution about Not provided keyword data, i have tried many tricks but still didn't get proper solution. Now I'll try your tips which you described in this post.
Reason we cannot compare webmaster data with goals or other dimensions is that Country and Google Property are the dimensions from Webmaster tools. They are not really integrating webmaster and analytics, they just added a section for webmaster in analytics so that user can see webmaster data in better reporting format with nice tables and graphs.
install the Google non SSL search plugin for Firefox - search Google with this plugin will pass keyword data to analytics https://addons.mozilla.org/en-US/firefox/addon/no-to-google-not-provided/
Awesome post, Tim! Had no idea about the site search feature (#3)...
Ranking are increasingly personalized, keywords exact match organic traffic is declining, deep pages are interestingly more targeted by algorithms: so I prefer the first methodology by tagging landing pages and monitor topic traffic
Great tips. At the end of the day webmasters are still able to get the organic data they need without it being explicitly given to them in the keyword area. By using the above methods you can still get a strong understanding of how visitors are finding your site, and what they're most interested in - which is the whole goal of seeing keywords in the first place.
Thanks for sharing it. Good one. If we know what is most searchable keyword phrase on our site and what are long tails search query on google(from which we are getting traffic), then we have just optimize the webpage to increase organic traffic.
You really have to get creative now to get around that dreaded Google road block. I still like to see what internal content is driving direct entrances into the site though.