Sifting through server logs has made me infinitely better at my job as an SEO. If you're already using them as part of your analysis, congrats - if not, I encourage you to read this post.
In this post we’re going to:
- Briefly introduce a server log hit
- Understand common issues with Googlebot's crawl
- Use a server log to see Googlebot's crawl path.
- Look at a real issue with Googlebot wasting crawl budget and fix it.
- Introduce or reacquaint you with my favourite data analyzer.
It’s critical to SEOs because:
- Webmaster tools, 3rd party crawlers and search operators won’t give you the full story.
- You’ll understand how Googlebot behaves on your site, and it will make you a better SEO.
I’m going to casually assume that you at least know what server logs are and how to obtain them. Just in case you've never seen a server log before, let's take a look at a sample "hit".
Anatomy of a server log hit
Each line in a server log represents a "hit" to the web server. The following illustrations can help explain:
File request example: brochure_download.pdf
A request for /page-a.html will likely end up with multiple hits because we need to get the images, css and any other files needed to render that page.
Image credit: Media College
Example hit
Every server is inherently different in logging hits, but they typically give similar information that is organized into fields. Below is a sample hit to an Apache web server, and I've purposely cut down the fields to make this simpler to understand:
50.56.92.47 - - [31/May/2012:12:21:17 +0100] "GET" - "/wp-content/themes/esp/help.php" - "404" "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)" - www.example.com -
| Field Name | Value |
| IP | 50.56.92.47 |
| Date | 31/May/2012:12:21:17 +0100 |
| Method | GET |
| Response Code | 404 |
| User-agent | Mozilla/5.0 (compatible; Googlebot/2.1; +https://www.google.com/bot.html) |
| URI_request | /wp-content/themes/esp/help.php |
| Host | www.example.com |
In reality, there are many more fields and a wealth of information that can only be gained through web server logs.
Googlebot crawl issues you can find with logs
Specifically in regards to SEO, we want to make sure that Google is crawling the pages we want to be crawled on our site - because we want them to rank. We already know what we can do internally to help pages rank in search results, such as:
- Ensure the pages are internally linked.
- Keep important pages as close to the root as possible.
- Ensure that the pages do not return errors.
This is all typically standard stuff and you can get this information easily without server logs, but I want more, I want to see Googlebot.
I want to look for Googlebot specific issues like:
- Unnecesary crawl budget expenditure
- Page it considers important / not important
- If there are any bot traps
- Is Google making up 404 errors by trying to make up URLs (think JavaScript)
- Is Google trying to fill out forms? (Yes, it happens)
Using server logs to see Googlebot
Step 1: Get some server logs.
Ask your client, or download a set of server logs from your hosting company. The point is to try and capture Googlebot visiting your site, except we don’t know when that’s going to happen – so you might need a few days worth of logs, or just a few hours.
To give you a real example:
Example domain has a PageRank of 6, DA of 80 and receives 200,000 visits a day. Their IIS server logs will amount to 4gB a day, but because the site is so popular, Googlebot visits at least once a day.
In this case, I would recommend a full day worth of logs to ensure we catch Googlebot.
Step 2: Download & Install Splunk.
Head over to https://www.splunk.com, sign up and download the product – free edition.
Note: the free edition will only let you upload 500mb per 24 hours.
Step 3: Adding your server log data to Splunk
I would recommend that you put your server logs on you local machine to make this process nice and easy.
I've put together a quick few screencasts, I know they sound cheesy, but whatever.
Step 4: Only displaying hits containing Googlebot as the user-agent
Step 5: Export to Excel
Simply click on the Export link and wait for your massive CSV to download. (Note: If the link doesn't appear, it's because the search isn't finished yet)
The Analysis, problem & the fix
The problem
Every time Googlebot came by the site, it spent most of it's time crawling PPC pages and internal JSON scripts. Just to give you an idea of how much time and crawl budget was wasted, please see below:
The real problem is that we had pages on the site that hadn't been indexed, and this was the cause. I wouldn't have found this without the server logs and I'm very grateful I did.
A look into my Excel spreadsheet
How to confirm what you're seeing is actually Googlebot
It's possible to crawl or visit a site using the Googlebot user agent, and even worse - it's possible to spoof the Googlebot IP. I always double check a list of IPs to what I see in the server log report and I use the method officially outline by Google.
How did I fix this?
1) Crawling PPC pages
I checked that these pages weren't indexed or receiving any traffic first, then I used robots.txt to block only Googlebot from these pages. I was very careful about this since I wanted to make sure that I didn't block Google Adbot (the robot that needs to crawl PPC pages).
User-agent: Googlebot Disallow: /*/cppcr/ Disallow: /cppcr
2) Infinite GET requests to JSON scripts
This was just another simple robots.txt block because Google didn't need to request these scripts. Googlebot basically got caught in a form, over and over again. Realistically, there's no reason for any bot to crawl this, so I set the user-agent to all (*).
User-agent: * Disallow: /*/json/ Disallow: /json
Results
I'm pretty happy to say that a week later, there was an increase of 7,000 pages in the index as reported by Webmaster tools.
Rand wrote about some good tips to prevent crawling issues, so I recommend you checking it out, as well as special thanks to the folks at ratedpeople.com for being kind enough to let me analyze and experiment on their site.
Additional resources
- Splunk documentation
- Apache server log documentation
- Enabling IIS logging
- cPanel server log downloads
- Differences between hits & pageviews
Feel free to follow me on Twitter @dsottimano, don't forget to randomly hug a developer - even if they say they don't like it :)
We usually always request a few days of server logs when we carry out a technical site review. There's *so* much value to it.
As you've pointed out - you can find thousands (if not tens of thousands) of requested URLs that never end up visible in the main index. Particularly, GoogleBot's new found love of crawing weird ajax / js URLs.
Proper log file analysis is easy - and you learn more about real SEO in a day or so of deep diving that you do in months of reading blog posts.
Right on.
And maybe what we commonly assumed may not be correct about Googlebot right? Looking forward to your post soon ;)
Working on it!
curious - did that post ever appear? Server logs is my current "must research" it is great to start to from other peoples observations!
Great post and great idea to trawl the logs to analyse the Crawl Budget :)
I have just few small comments / clarifications:
The image "Request for 1 web page" with multiple arrows beling shown as responses should in fact show multiple requests (for the one page) with the multiple responses. I think you are explaining this below the image with "A request for page-a.html will likely end up with multiple hits because we need to get images, css.. " - I would just like to clarify that "multiple hits" are multiple pairs of "request/response" messages rather than one request message with multiple response messages which is what the picture shows.
The "Keep important pages as close to the root as possible" - this presumably refers to clickpath (from the home page) and not to server file path. I could have file server path 5 levels deep, but if the URL is linked directly from the home page, its click path is only 1 level deep and therefore the page is emphasised within the information structure.
With regards to disallowing cpprcr folder - this can be done at the User-agent: * level since Google Adbot ignores blanket exclusion under "User-agent: *" Unless, of course, you already have a separate section in robots.txt for Googlebot, in which case it should go in there.
And lastly, if you do have separate section for Googlebot (or if you have added a separate section for Googlebot to your robots.txt as per paragraph above) then the javascript /*/json disallow must be added to Googlebot section in robots.txt since if there is Googlebot section in existance, Google will ignore blanket user agent section. (i.e. if you have User-agent: Googlebot in robots.txt then Googlebot will ignore anything under User-agent: * section)
Hey,
1) Yes, you're right and I thought about making that clearer. "multiple hits" are multiple pairs of "request/response" messages rather than one request message with multiple response messages which is what the picture shows.
2) Yes, keeping important pages as close to the root is representative of click depth from the homepage. ex. www.example.com/the/file/path/doesnt/matter/ <==That file path is very long, but it doesn't mean it's necessarily buried in the site. We could in fact link it from the homepage regardless of how many folders you see. Sorry if I confused anyone!
3) I'll have to try that sometime - I didn't know that. Also, I wanted to be super careful I didn't mess with Adbot, therefore I used the specific user-agent.
4) Yes, great tip. I had to do quite a bit of testing to find that out, and I did in fact specify this in the Googlebot specific section of the robots.txt file as well.
Great feedback, I feel like I should pay you or something ;)
Great post!
Another thing that you should look out for is secure counterpart URLs. Googlebot fetches https://www.example.com and then it might fetch https://www.example.com:443 and that could lead to duplicate content issues. I have noticed this on a lot of sites where I have installed SEO Crawlytics plugin.
Great tip, should have included that. Thanks Yousaf.
Thanks for the contribution Dave. Information its reallly valuable. Great post; great author.
I thought that "splunk" was something else before I read it again...
Well done man.
+1
You and everyone else who's heard the name ;) Great product though...
I <3 Splunk! I've been using it for over a year now ... it's totally awesomesauce! I like to keep a macro of the Googlebot IP ranges so I can accurately filter our good bot traffic.
Hi Dave,
thanks for the post, you have demystified server log analysis (I am sure there is more that you can do with it).
Question for you, I have read a fair amount about not blocking user agents in the robot.txt file as that becomes a veil beyond which Google can't see and doesn't know anything about. Also, it does nothing to pass page rank and so you can lose value into a black hole (if I understand things right), is that something that you are concerned about? Any suggestions for addressing it differently or at the root so that you don't have to resort to just blocking it via robots.txt? Am I making to big of a deal out "issues" of using robots.txt in this manner?
Thanks again for walk through. Cheers!
Hey,
I actually barely scratched the surface of what you can do with server logs, but I tried to keep it short so I didn't lose reader interest.
In regards to your question, I wish I could give you one simple answer. Firstly, there's nothing wrong with blocking Googlebot or any other user-agent in robots.txt - the point is to stop them from crawling pages you don't want / need them to see. There's a bunch of other reasons as well, maybe a specific user-agent is aggressively crawling your site and racking up hosting bills, and you just need it to stop.
As for PageRank, the theory is that Google won't pass PageRank to blocked pages. So, losing value is a concern - but it depends on what you're blocking in the first place. A site like bbc.co.uk actually blocks their /search/ folder - which in my opinion might be a mistake (because Googlebot can't spread the external link juice around the site). In this specific post, I blocked weird PPC pages that weren't linked to, and javascript URLs that were part of forms. There is no problem with that at all.
Ideally, if you make the *perfect* site, you won't need to block anything. Sometimes you just need to, and I think Rand summed it up nicely in his last 2 points in this post https://www.seomoz.org/blog/headsmacking-tip-13-dont-accidentally-block-link-juice-with-robotstxt.
There are so many considerations you need to run through before making blocks, using nofollows and other methods of controlling Googlebot's crawl path. Each scenario has it's own perils, but it shouldn't stop you from making changes.
Does that help?
Wow David!
Thank you so much for sharing Splunk in this post.
I've always loved my logs, but you just made my dark fascination for digging into them something I can do in a flash on company time!!
Sha
Dave, very good insight on how to dissect Googlebot crawling activity on a website.
I can't describe how useful and important it is to keep an eye on your crawling reports. We were recently saved from a big mishap on one of our online shop because of our crawl reports. The website had:
(i) numerous 301 redirects from URLs which were outdated and still happened to be on the site, and
(ii) crawling of 2 or 3 different query string URLs of a single page thereby creating duplicate page error along with consuming valuable crawl budget.
You won't believe that even the advanced crawling software's that we have employed to monitor our website for all on-page errors were not able to identify these issues for us. As soon as we noticed them in our daily crawl report, we were able to take precautionary action of changing the old URLs to newer ones, putting rel=nofollow for URLs with query strings and placing self-referencing canonical tag on few important pages.
We don't completely rely on server logs. We have our own custom PHP utility which generates a daily crawl activity report for all major search engines. You can play around with the data once you have it in your database. Our script monitors Googlebot, Bing, Yahoo, other major search spiders including mobile bots for mobile websites. (btw, we have seen a rapid increase in sales through mobile website - is it a paradigm shift underway?)
Due to NDA, I am unable to share the utility which we have developed, however sharing here few of the articles that my development team has referred at the time of implementation:
https://stackoverflow.com/questions/916147/how-to-identify-web-crawlers-of-google-yahoo-msn-by-php
https://www.user-agents.org/
https://www.liamdelahunty.com/tips/php_gethostbyaddr_googlebot.php
As it goes, you can't build a bug-free software, and hence its highly recommended to test your implementation thoroughly and keep it updated all times.
Its nice to see the way you have given justice to this complex, but significant area of "crawl" monitoring.
Thanks for the insightful comment and the resources you left. Sounds like you and your team have managed to completely automate the process, which is incredibly efficient of you.
Thanks for sharing, much appreciated.
Great post, I regularly check my server log to see the visits and ban the spam looking IPs. Theanks for the tip that it can be used in this way as well.
Hi Dave,
Great post!
One other way to do this is to use the Web Intelligence App (free download on Splunkbase) and use the pre-built reports. There is a pre-built report for bot activity, with the ability to drill down into the raw logs or search on the individual activities.
Disclaimer: I work for Splunk.
I was hoping one of you guys would stop by ;) I know I've only scratched the surface of what Splunk can actually do, but thanks for the tip.
In fact, I think you should write a post for SEOs. I'd love to contribute so feel free to private message me.
Excellent information Dave. And thank you for providing the screenshot video.
Question: in your logs, are you finding the same crawling issues with Bing or other crawlers?
Bing who? Just kidding. Catching Bingbot turned out to be harder and definitely crawled less of this particular site, so I didn't actually look close enough.
I've probably made a mistake by not checking Bing, but I definitely saw the same pattern with Yandex.
Np, and thanks for reading.
I suspect following through on this might solve an ongoing issue we are having with the indexation of a client site (600K pages, had 260K indexed at one stage, now just 15K). Given the importance of the client, however, I may ask the technical team to report on it before I begin dabbling in things I don't understand completely.
Thanks, Dave, for the advice. It certainly gives me a new avenue to explore with this project. Very much appreciated. Also, good work presenting a tricky topic in such an accessible way, I'm sure that will be a major plus for a lot of people.
Thank you for reading and leaving such a nice comment!
Great post, we've got someone from the team checking this out now on a client's site. So the extra pages that you got indexed, how much extra traffic organic delivered? If even only a quarter pulled in a single extra visit a month, that's a return of 1,750 visits, all for a couple of quick robots.txt fixes. Wow.
I don't have enough data at the minute, but why don't you bug me on Twitter in 3 weeks time and I'll share some results? @dsottimano
Thanks Dave!
Interesting post, very useful information. (as always)
I do want to point out that some of recorded Googlebot visits are probably belong to Googlebot impersonators.
Just recently our company (Incapsula) completed a Googlebot case-study (covering a group of 1,000 websites) in which we've discovered a 21% avg. Googlebot impersonation ratio. Some of these were SEO tools (using Googlebot user-agent is a good idea if you want to do this type of crawl) but ~16% were malicious and used for some kind of cyber-attack.
If you are interested, you can find more here: Fake Googlebot study
For this reason I also wanted to share Google official verification method and also say that its currently insufficient as Google dose not keep/update IP ranges. For example there is a Google Image bot that visits from Chinese IP and this can look suspicious (and lead to blockage) in the absence of official information.
Wonderful post, thank you. I better be checking my server logs now :-)
Great post - I'm just scratching the surface of 'technical' SEO and this is just the sort of thing I want to incorporate into my site reviews.
I unfortunately came across a hurdle before being able to dig into the data: I cannot verify the 'clientip' as being one of Googlebots... the IP I have is 10.248.57.207 - MXtoolbox tells me ' 10.248.57.207 is a private IP address' when trying to do a rDNS lookup. Do you have any further information about how to find out what this means and what possible problems I might encounter?
Thanks :)
Probably not the real Googlebot then. Anyone can easily change their User-agent to mimic Gbot, and it looks like this is the case. If that IP appears in your server logs daily, and is causing your site problems (crawling aggressively, consuming too much bandwidth etc..) - I would block it from accessing your site. Once blocked, and you see another similar IP doing the same thing, try guiding that IP into a bot trap.
Hi,
Thanks for such a informative information.
I am having one big problem here. I am facing some unusual problem from Google Bot.
"My subdomain pages are get Indexed in Search Engine but when I check the Cache but it is not CACHED. Showing 404 error."
This is first Time I am getting such wired experience from Google.
Please suggest some solution.
Thanks in advance.
Indexed doesn't necessarily mean cached ;) I don't know what the rule is, but Google decides whether the page is worthy of being cached. Unfortunately I don't have the fix, but just make sure you don't use the noarchive tag (<meta name="robots" content="noarchive">) and that the page content is not duplicated anywhere else.
Is it that big of a problem if the page is ranking? I'd probably say no and don't worry about it.
Analysing your server logs is of vital importance. There are various tools out there. I suggest Apache Logs Viewer. It's free and you can easily filter and sort out the log data easily.
You can also produce some nice reports to hand to your clients/supervisors. More information from:
https://www.apacheviewer.com
Great post. I'm not sure if something changed of late. If you're running PPC on your site and happen to block the Googlebot like the example above, you'll get notification from Google saying that Google's adsbot are blocked on your site and that your ads cannot be approved even though the robots.txt tester within GSC shows as 'allowed' for adsbot-Google.
So, the solution is
User-agent: Googlebot
Disallow: /*/cppcr/
Disallow: /cppcr
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
Allow:
this above steps look time consuming and manual... i am using a automated log file tool at botify.com which is half the cost!
Hi David! This is such a very helpful post. Thank you very much for sharing on how to use server logs to see Googlebot. I will definitely try this. I will also be sharing this to my friends.
This is a brilliant guide and is only let down by Splunk being such a pain in the ass! Not the fault of the author of course.
I wonder how many sales they hook with this page https://www.splunk.com/view/pricing/SP-CAAADFV
Am wondering about how the bot makes up files that aren't on my site? I run a simple 404 snippet that emails me about missing pages, missing images or hack attempts and I am constantly seeing googlebot making pages up.
Is the bot making suggestions on what files should be created for the site based on...? The site I recently built gets a few requests per day for pages that would seem to be a good fit as far as naming goes, but I know my sitemap is clean and definitely those pages do not exist So ... what should I do? Create the pages or do a redirect for a ghost page made up by the bot?
PS. the little 404 app is simple. the ErrorDocument 404 line in htaccess leads to an exact copy of my index page: example: index2.php - it opens when someone attempts to go to a page that does not exist as well if something is missing on an existing page. Any problems I receive an email telling me what file was requested and what the IP address is that requested it. Also good if you are selling ebooks and you want to know when someone has reached your uber-secret download location.
Makes it easier to IPBan those looking for a backdoor - as well, the little app is great for a new build as it instantly sends feedback of anything that is missing. If anyone wants a copy hmu.
Dave, Amazing job and thanks for your valuable thoughts on this topic.
Hey Dave,
I believe the example you are showing isn't a valid Googlebot IP. 50.56.92.47 Appears to come from Rackspace's servers. I have found that 99% of the time (based on the log activity I have analyzed) Googlebot IP's start with 66.249 so that is an easy way to quickly filter out bots pretending to be Googlebot (or get their IPs to block).
Good catch. Actually, you can spoof Googlebot's IPs and it's important to use Google's official method that I've outlined above title: How to confirm what you're seeing is actually Googlebot.
Dave, sorry if I missed it but what columns did you use for both axis on the graph? And is that a pivot table graph?
Are there any other columns you would recommend plotting for comparison?
Excellent post, thanks!
Nathan
Hey, so the graph above is based on a pivot chart. In my spreadsheet I had a field named "root" whereby it took the first folder in the file path. Example: example.com/cppcr/abcd/efg - the "root" would be "cppcr".
In the pivot chart sheet:
My column label: Useragent (filtered to Googlebot)
Row label: Root
Values: Count of URI
I think this was by far the easiest for me, but there are so many different ways of looking at this. I guess another important graph would be: count of URI on your site, and how many Google actually crawls in a month.
Good luck!
that sounds very useful for sites with PPC and/or affiliate campaigns.
Great post Dave, how ever I'm going to PPC and affiliate campaign, I hope it must have ++ advantages for me..
Thanks a lot.
Manoj
Great tips on some good tools - we've been looking at dashboards recently and have been impressed by the power that splunk opens up. Awesome.
Wao very good information for all seo expert Thank you so much Dave
Really too much informative post on Googlebot and how it is working and how we can use it and can take the benefits from it at huge level.
Thanks for sharing i always wait your stuffs
Great to know about this, really Dave your contribution is very valuable and I am thinking to use it ASPS.
great post Davide,
i have started quite recently digging into server logs, because i found out it was practically the only way i could follow to really understand why googlebot was indexing errandly important product pages of one of my clients.
here you cite Splunk, but i imagine you worked also with other log analytics tool. if it is so, would you like to tell me what were they?
I like the paid version of https://www.weblogexpert.com/ and I've also used https://awstats.sourceforge.net/ before on different hosts (didn't get as detailed as I wanted, but I probably didn't use its full potential). So far, Splunk is my favourite and the quickest.
This blogging seems very informative blog....