This tool is used to test how accessible your site is to search engines and can help you quickly diagnose potential crawling issues and give you an overview of your site's search friendliness. The tool will spider the URL you enter as well as all the internal links on that page (max 50 per report). For each spidered URL, it will examine the following: whether it's indexed in the major search engines, last time google spidered the page, http status code, primary keywords on the page, meta description, and the number of internal links on each page.Since then two things have changed: The Crawl Test no longer uses the Yahoo! Term Extraction API to retrieve relevant keywords, instead it uses data from our in-house Term Targeting Tool. Secondly, unless you are a Premium Member you can only run one report per day and each report will only spider 5 pages. Premium members can run as many reports as they like and it'll spider 50 pages. I'd also like to point out that a good way to test the spider-ability of important pages on your site is to run a crawl test on your site map.
I've got a sample report available that I ran on our Web 2.0 Awards. As always, suggestions and ideas are welcome.
Handy tool. I really must get round to getting a premium membership... ;). Having a crawl test history would be quite good though as well =)
This is definitely a powerful and useful tool. It's what got me to sign up as a premium member.
Your new tool will keep me as one :)
Keep up the good work guys!
Excelent tool, makes me want to sign up for the premium membership.
Thanks for making it public..
Gave it a try and it didn't work correctly in my case. It looks like you're using the site: command to get your index count but this is inaccurate as you have to click the "repeat the search with the omitted results included" link to get all the indexed pages.
In my case it said 4 pages indexed when in reality there are about 13,600..
Good point - we should see about trying to retrieve both pieces of data for the crawl test. Thanks for the great idea!
Keep up the good work!
Fabbo tool and judging by the comments, useful beyond measure to some!
Great tool, but I'm struggling a little as it seems to be dropping my domain name on site root relative links. Is this just happening to me?
I'll take a look, thanks for the heads up.
i am not a member :(
but if i run the report today, il will give me 5 pages... if i run it tomorrow, will i have another 5 pages? or same 5 pages?
thanks
The same five pages, assuming you're running it on the same URL.
You know I think the tool just helped me figure out an issue on my site that's been puzzling me for a couple of months. I'll have to make a couple of changes and run another report to be sure. If it did I owe you some thanks.
Really useful tool Matt - awesome of SEOmoz to provide these useful tools to everybody.
2 thumbs up!
Great idea on the sitemaps... truly one of those "why didn't I think of that" ideas.
Going public will also be good at getting a lot more feedback on any quirks or issues!
I was thinking the same thing about the sitemap. Seems so obvious now that I hear it.
Very cool tool. These may definitely push me to get the company to pay for a Premier Membership. I love that you guys are developing very useful tools. Congrats!
Excellent tool - cool way to find dupe description tags for one.
i was ectually allready using it becuase i really like the commercial articles, and really liked this tool. so great you guys put it publicly great work!
Its one of these tools which make me wonder why i dont' have a premium membership yet. Great job and works like a charm
Thanks for the sitemap tip, hadn't thought of that and it works great!