Over the past couple of years, we've seen a number of new web-based SEO crawlers (including our own Pro App here on SEOmoz). They're great tools and can provide a lot of insight, but sometimes you've just got to get down and dirty into a big site, and you need a desktop crawler application. For the past few years, that's pretty much meant Xenu's Link Sleuth. Recently, a new entrant has thrown its hat into the ring – Screaming Frog SEO Spider.
I was amazed, on social media, how many people still hadn't heard of Screaming Frog. Actually, a few people still don't know about Xenu. So, I thought it would be a good time to put both programs through their paces.
Xenu's Link Sleuth
In this corner (that doesn't really work when you can't see me pointing, does it?), we have Xenu, a PC-based desktop app that bills itself as a tool for checking broken links. Over the years, enterprising SEOs have found it's capable of a lot more. For such a popular tool, it's funny how little we know about the creator, a German [engineer?] named Tilman Hausherr. All I can tell you is that he wrote Xenu because he was bored, he doesn't want your money, and he really dislikes Scientology.
.gif)
Pro: Xenu is Free
Let's start with a few pros and cons. You really can't beat that Xenu is completely free. We all owe Tilman a hearty "thank you" for putting it out there over a decade ago.
Pro: Xenu is Fast
While a large-scale site can take a while to fully crawl, Xenu is a machine, and it's an impressively multi-threaded app, considering when it was built. It can power through 1000s of pages pretty quickly.
Con: It's Not Intuitive
Xenu really takes some getting used to, and it's clearly built by an old-school software designer. The basics are easy enough, but getting into the real SEO applications of Xenu takes a bit of a learning curve. To be fair, it wasn't built for what most of us try to do with it.
Con: No Mac Version
This is the biggie for many people – Xenu only runs on Windows PCs. There's no Mac or Linux version, and the author doesn't plan to release one.
What Can Xenu Do?
At its core, Xenu really is just a link checker, churning through your site to test internal and (optionally) external links:
Since Xenu operates like a crawler, though, it can really help test crawl paths and find holes in your internal linking. Xenu also reports useful stats, like the crawl level, outbound links from a page, and inbound links to a page. It also returns TITLEs and META descriptions, which can be handy at-a-glance.
One really useful feature in Xenu is that, once you find a broken link (page or image), you can just right-click on it, select "URL Properties", and you'll get a pop-up like the one below showing you all the pages linking to that broken file:
Xenu is a work-horse, and it's still a great tool for churning through links fast. It's only real fault is that we've tried to push it beyond what it was designed for, and it wasn't really designed for SEO.
Screaming Frog SEO Spider
That's where Screaming Frog comes in, a desktop crawler specifically designed for SEOs by a UK search marketing firm. Because it's built specifically for SEOs, it has a lot of bells and whistles that Xenu doesn't. Let's start with the quick list of pros and cons.
Pro: It's Built for SEOs
We'll dive into this one below, but Screaming frog has been built by SEOs for SEOs, and that fact is obvious as soon as you launch the application.
Pro: It's User-friendly
I found Screaming Frog very easy to dive into. The advanced uses will take some time to discover, I'm sure, but any experienced SEO should be able to get up and running on it quickly.
Pro: It Runs on Your Mac
Screaming Frog supports both PCs and Macs. If you have a Mac shop, that's a big one.
Con: It's Not Free
Ok, here's the down side. It's £99 per year (that's about $162, currently) to run the full version of Screaming Frog. The free version will only crawl 500 URIs. The free version is nearly fully functional, though, so you can get a good idea if it's worth the price for you.
What Can Screaming Frog Do?
On the surface, Screaming Frog doesn't look too much different than Xenu:
Like Xenu, Screaming Frog tracks the crawl level, internal links, outbound links, TITLE tags, and META descriptions. It also adds:
- HTTP status codes
- TITLE tag length
- META description length
- H1, H2 tags and lengths
- Canonical tags
Many of these elements are also broken into their own tabs at the top, so that you can easily focus in on problem areas. You can right-click on any URL for many more options, including checking indexation in Google, Yahoo, or Bing and checking backlinks in Open Site Explorer, Yahoo Site Explorer, or Majestic SEO:
The paid version includes one incredibly useful feature – the ability to custom filter and search HTML source code across a site. Digging into all of the features is beyond the scope of this post, but Branko Rihtman has a nice Screaming Frog review over at SEO Scientist. If you want to dig deeper into Xenu, Tom Critchlow had a good Xenu recap here on SEOmoz last year.
And The Winner Is...
Sorry, I just can't do it. I've loved Xenu (the app, not the galactic overlord) for years, and even if it weren't free, I'd have gotten my value out of it many times over. I'm excited about Screaming Frog and what it can do, though, and I'm especially excited to see new products and innovation finally coming to desktop crawlers. This is a win-win for technical SEOs everywhere, and I'm looking forward to the evolution of both desktop and web-based crawlers.
Hi Peter,
it is somehow funny that you wrote this post about Xenu and Screaming Frog this week, as I rediscovered both this week due to audit jobs; maybe I was "incepted" by your question on Twitter.
I have to agree with you about the "not declared" winner. Even though Xenu Sleuth is surely a powerful tool, maybe the fact it was not really created for internal SEO analysis purposes makes of it a little bit harder to learn. I use it a lot, especially after the last year Tom's post about it and thanks to all the posts Distilled dedicates to it (last, but not least, this recent post by Mike Pantoliano about Site Structure Audit).
But if you need something that delivers you a very friendly and complete output, then Screaming Frog is the right choice. It has a cost (but it is a one time payment and not so high), but if you have to check up small sites (under 500 pages) you can use it for free.
Finally, I won't forget a third alternative: IIS SEO Toolkit. Powerful and multilingual, even though should have to update the suggestions it gives (still refererring to somehow dated factors' importance). Richard Baxter and Richard Butler have dedicated some interesting posts to it on SEOgadget (I suggest you to start from this one).
P.S.: and yes, the SEOmoz Crawler is fantastic. Shame you have to wait sooooo long before it delivers you a crawl (looking at the SEOmoz devs team)
Trying out the IIS SEO Toolkit has been on my list forever, and I don't have any good excuses. I need to start working with that more.
The IIS SEO Toolkit is fantastic. I highly suggest you try it. It could use an update but even still it smokes the competition..oh and its free.
Never heard about the IIS toolkit, thanks for the heads up. Looks really....extensive.
Aww - thanks for the shout Gianluca! (as always!)
IIS SEO Toolkit is awesome - and a huge mention to the data export on the pro app. Download that data by csv people! It's very handy.
Hey Pete,
Thanks for the write up about the SEO spider :-)
We have used Xenu for a long time and love it still to this day. It just didn't quite do enough for us from an SEO perspective as time went by and hence we built the SEO spider.
Originally this was just built and used by us internally for our SEO projects, but we decided to release it and see that others thought as it saved us such a significant amount of time. It's fairly well known in the UK, but has taken a little longer to get picked up in the States.
Anyway, appreciate the post. If anyone has any feedback, love to hear it. We have a very long development list of features to add to the tool, so keep an eye out as there is much more to come.
Cheers.
Great job Dan!
I've fully moved over to the paid version of Screaming Frog, and the team have been great at adding little suggestions - so thats putting it ahead of Xenu.
Ironically our corporate firewall was what prevented me using Xenu to start with, as newer ones falsely claim its a virus! Hence my need to try out something new, and luckily found Screaming Frog - a great tool!
Still - thanks to Xenu for those good times we had...
Cheers Rob.
Anymore suggestions, you know where I am :-)
Got loads in the pipeline, just takes time to work through them!
Its happy to see the Screaming Frog Developers itself is coming back with comments.
Congrats Development Team for this pretty useful tool for Onpage SEO Audit. After analyzing screaming Frog it sounds as a really helpful and Screams aloud.
I know If a software is worthy to getting mentioned in SEOMoz blog, then in turn it will famous to the SEO Pro’s around the globe. Thanks Screaming Frog for the wonderful product.
Thanks for the kind comments Jackson.
The spider was actually written by a single developer (similar to Xenu again!) :-)
We are a small team!
Cheers.
Thanks for chiming in on the post - love the tool so far, and look forward to where you guys are headed with it. As entrepreneurs, it seems like the best tools are always the ones we build to solve our own problems.
Dan
It is nice to see this. Using Screaming Frog felt like someone was trying to answer the limitations of Xenu, and it is nice to read that you guys were fans too. One thing that Xenu does have and I have not been able to figure out in Screaming frog is the ability to do crawl password-protected pages. I was working on a page on a dev server, and had to revert back to Xenu.
Just downloaded the Demo-Version.Might be a paying customer soon.
Although I use Screaming Frog on a weekly basis, I feel someone should mention WinWebCrawler. They have a free 30-day trial and have been around for quite some time.
Also, I rely heavily on AnalogX now, it's a great FREE broken link scanner with many great features.
thank you! just tried AnalogX LinkExaminer and seems great :)
There is a great link checker for mac that is free (or maybe open source, cant remember)
https://peacockmedia.co.uk/integrity/
Super fast, super reliable, super accurate.
Well, i've used Integrity during the past months and it has the tendency to become unstable... but it's still free.
That's great, I didn't know about this toll and I'm testing it right now. Fortunately my main site didn't have any errors, but I did find that I should do about my internal linking strategy. Some pages that have hardly any value to the search engines get a lot of links.
Now I just have to find a way to do this better.
I can second the notion that Integrity is pretty good. The ironic thing is... It doesn't have much integrity and can sometimes become really buggy. But for a free tool, it's pretty good.
Excellent post. I switched from Zenu to Screaming Frog shortly after a reading post that I believe Rand did covering some of the Top SEO tools. I was blown away about how much easier it was to use and that it gives amazing visibility to other key SEO factors you would want to review in an audit. Definitely improves productivity when audting sites!
I am amazed there isnt more buzz about this great tool. I highly recommend it for any inhouse SEO's - both large and small - the paid version is worth every penny. I help support several large, complex ecommerce sites on SEO and when I use this tool, leveraging filters/settings properly within the tool, its amazing how well it can handle these kind of sites.
Their support has been fantastic so far as well (just like SEOMOZ's!). I did not know what to expect but was pleasantly suprised on how professional, friendly and responsive they were. My only wish is that it could save and store the report data after it is run. I have had a couple instances where my computer has installed those darn Windows updates without my knowledge and restarted after the tool has been running for hours. Once this happens, all data is lost and you have to basically start over. Also- it would be great to have the history stored so you can see if the improvements you make from using the report data in this tool are working. That's one of the things I love about SEOMoz's crawling tool. Having the history and being able to show visual charts on increases or descreases in 404 errors, missing titles, etc has been very valuable when I have to report progress to mgmt. Sometimes it takes awhile before the impact of SEO improvements show up in traffic numbers so, being able to visually show the progress of cleanup helps mgmt understand that things are happening and to hang tight for those improvements to start reflecting in traffic and sales. Without this kind of data, their perception is that nothing is being done because traffic #'s are not increasing right away as they expect.
I am a huge fan of SEOMoz and Screaming Frog Tools and so thankful that companies like yours and SF are investing in development of SEO tools that are reliable, help improve productivitiy, and are designed with usability in mind. As a corporate SEO, it makes my life easier when training others in my organization who are "non-technical" on how to leverage these tools. Now, less reporting & monitoring falls on me which allows me to focus more time on the more complex technical SEO issues on our sites and web platforms. Both of you, please keep up the great work!
Thanks Susan, appreciate the comments. Great to hear it's making your life easier :-)
I am not sure if I mentioned this when we spoke (via our support), but we are working on a save project/crawl type feature that you can re-upload back into the tool aswell as a failsafe to avoid losing data during unexpected crashes etc.
Any other suggestions, do just let me know.
I have used the full version for over a month now and found it briliant. Features that I would like to see added is a way to control a number of threads the Screaming Frog uses and the way to contro the number of reqests per second.
Both on the list. We haven't started working on speed yet really, so plenty of room there!
I second the request for a project save feature. I have the paid version and many of the sites I audit are enterprise level 20k+ in size. The tool is awesome, just want to make sure i can save and come back to the data as needed. Is there a timeline for when this feature will be released?
I can't give you a date, all I can say is - We are working on it right now. So, not too long I hope :-)
Good post Pete.. But Still my winner is Xenu.. I consider it be one the excellent Freeware helpful to all!
I consider Xenu as one of the best Freeware software and a must for a Online marketing professional. Have you tried Web Data Extractor, something similar, but not sure it is as good as Screaming Frog SEO Spider. But the issue was that too is paid one. Anyway will try out the Frog SEO Spider..
I feel something weird with the name ‘Screaming Frog SEO Spider’, but that may be the differentiating factor the company looks into for marketing it!
Anyway thanks for Post!!
Thanks for this post Pete, I've been looking forward to it. After I changed to a Mac, I freaked out a bit when I realized there wasn't a Mac version of Xenu. I asked on Twitter if there was anything similar to for the Mac and no one knew of anything. I'm definitely going to be giving Screaming Frog a try. As much as I love Xenu, I really dislike having to go back to the PC just do use that app. :)
J-Lo you could run a virtual machine... Xenu is worth that much :D
I love A1 Site Analyzer for this kind of stuff. It even has a little on-site PageRank flow checker to be sure your Copyright page isn't you top linked page. Very handy.
I'm an A-1 user as well. Xenu is great for a casual user, but A-1 does all that Screaming Frog does (at least as was covered here) and is cheaper. I love it!
Pete
you made my day with this post. I was very excited about this post. Dan has done a good job with the Screaming Frog tool and I am excited about the improvement opportunities that will result from more vendors producing these SEO Tools.
One of my dream tool would be the merger of SEO Browser and Xenu. Screaming Frog is heading in that direction but a couple of things need to be added to make it even sweeter. But I agree, no clear winner here. Xenu in my heart, Screaming Frog in my Head.
I love this stuff, great (x 5 ) post.
Although I love XENU... I'm going to +1 Screaming Frog. It's quick, detailed and the interface simple.
Took me about 2 days of experimenting before I bought it, if that says anything. The only CON I've seen so far: it can hang on large websites - the most I've been able to crawl is 10k pages.
However, I'm not sure if these are isolated causes (crappy connection, some fancy script on their server, etc..)
Yep, I'm a FF (Frog Fanboy)
Thanks Dave.
Maybe we should do a FF badge? ;-)
Dave,
I felt your pain at first as well. I use screaming frog on large complex websites and based on some great advice from SF- make sure you are leveraging the "Exclude" and "Spider" options in the Configuration section. By using this, you can actually set it to specific areas (smaller chunks of the site) that you can run. You can also turn off crawling of Javascript, images, etc and keep the crawler focused on pages. By leveraging both these areas, I have been able to get the tool to crawl 70k+ pages. Their FAQ section has some instructions on how to manage these configuration settings. Best of luck to you!
This is good news for me. I have several large sites 300k+ pages and while Xenu allowed me to specify a starting directory it didn't allow me the option of limiting to ONLY that directory. I could exclude urls that begin with xxxx...but that would mean manually inputing probably over 1,000 values to skip! And I end up having to run it three or four times and then having to start it over to skip certain directories that weren't necessary. So this is my question:
Can Screaming Frog allow me to specify a directory as an include only pages in this directory? If not....note to devs...I'm buying! I need to create sitemaps for several directories as many of them are larger than 50k pages. Anyone know of a better way to do this? Currently I'm working with a complete Xenu crawl of 300k urls, exporting to csv and then segmenting out the directories there and creating several url list .txt files. Then I have to use A1 Sitemap Generator which is one of the only ones I've found to import URL's from a file and output an xml sitemap file.
Also, can Screaming Frog generate an images or video sitemap??
answered my own questions. Looks like it can't even generate an xml sitemap. Let alone images or video. Also, you can't really limit the crawler to a specific directory. It will still crawl outside that directory if it finds a link. I was hoping this would solve my sitemap creation woes...still looking for a good tool.
Sorry, was late seeing your query.
The SEO spider doesn't generate sitemaps at the moment, but it's on the list. (Not at the top though).
You can specify a directory to crawl, but it will crawl onwards if it finds a link, hence you are correct - you would need to exclude any additional directories if you didn't want them crawled.
Again, that said - We are adding this feature aswell which is a priority. :-)
awesome! I'll check back in periodically to purchase once it does those things. Do you announce those kinds of things on Twitter? I'll follow you there for updates. Great to have you posting here in the blog!
Yep, we announce all updates on Twitter and on the blog aswell.
So keep an eye out, should have some big updates over the next couple of months. :-)
Going off the topic.. but faces reminded me scream movie :P
GSiteCrawler is a solid, free tool as well.
I use Xenu to find check broken links and no. of pages on a website that are otherwise hidden from search engines. I like to compare no. of Xenu pages to the no. of pages indexed by Google to find any indexing issues in a website.
I don't really trust xenu when it comes to checking server response though as the no. of request time outs it gives is more than actual (may be it's coz it's running on my machine). Its reports can freak you out if you check your server performance from this tool.
Xenu generated sitemap.xml also get a little messy.
So in all I use xenu:
I never heard of screaming frog; would love to give it shot now. Thanks!
Thanks for the extra tips!
both tools are great, but tonight i have to perform a SEO audit for a client and test Screaming Frog and this tool just blow my mind : simple and beatiful interface, the tool is fast and the report is easy to read even for non expert. screaming frog team : well done :)
Hi ,
I am a new bee to the Link Analyzers and for a use case we have, tried running both Xenu and Screaming Frog on a public website ,both are free versions and i see a huge difference in the result set of Xenu and Screaming frog.Is it cause of licensing? Can we set how deep it can go in screaming frog?
Also I need to crawl site that needs authentication, how to achieve this with Xenu/Screaming frog?
Please help.....
Great Comparison Dr Pete. Indeed, Screaming frog is the winner.
I definitley prefer screaming frog. I haven't had any issues with it as of yet and I love how user friendly it is!
SeoMoz web app reports broken link but doesn's say from wich page. Thanks, Dr. Pete for
this article. I just fixed 3 broken links on my site, reported by xenu.
I use both and on Mac's as well. Using Crossover you can open Xenu and run it. To be fair I'm having a hard time finding what Xenu does that Screaming Frog doesn't already.
Gary Beal
GaryTheScubaGuy
Screaming Frog is ok, it doesn't really seem to have anything new or interesting to be honest. If you didn't have screaming frog but you used SEOMoz generally you wouldn't be missing much.
an odd response - SEOmoz and SEO spider are way different - SEOmoz is not on demand, but part of a larger suite of tools. as a crawler tool seospider is way superior - and you should be using both!
Hello,
I used screaming frog to analyze our site perunature.com. It says that there are no metatags or meta keyywords but when i check the source code I see that they are present. Any ideas on why this is happening?
Best regards,
Jeff
All I need now is an affiliate deal with The Screaming Frog :o)That way I too can make money on that exceptional tool.
I am looking into the other tools mentioned on this site, and might even make a link to their reviews when they are done.
Thanks for an excellent post, as always
A long-time fan of Xenu, I must say though that it should upgrade if it wants to complete with tools like Screaming Frog. But just because it's good old Xenu, I wouldn't rule it out completely. I am sure there are people that are very used to it, who are comfortable with it and wouldn't give it up easily.
On a side note, as somebody already mentioned it's not just Screaming Frog vs Xenu anymore. There's SEOmoz site crawler, the new Webmeup crawler.. It's getting really crowded in the jar :)
Xenu is very helpful. I didn't realiza that I have almost 80 tys links internal & external. Lovely tools!
Thanks
---LeeRit
Great post
Xenu "returns TITLEs and META descriptions".
Xenu will show meta title tags but not meta description tags (I wish it did).
I like Screaming Frog, because it provides you information about the canonical characteristics, Header tags, meta refreshes, and everything you need to help you in site audits.
I asked a while ago, and they state that there is an option to switch the User Agent coming in the future, because a site scan from Screaming Frog UA may be different than a site scan from Googlebot.
Regards
Caraseobali
We've recently launched seopler.com and were selected for the Web Summit 2015 Alpga program. Our tool does everything XENU and Screaming Frog do and more. Also as our software is cloud based no local resources are consumed. Millions of pages can be crawled as an elastic server is run up for each job submitted. As one Web Summit attendee we demonstrated to quoted.
"Seopler is to SEO what Excel is to Accountants"
We offer a free account and would love feedback from the Moz community on our product.
Just tried to crawl a site with 400k pages, it crashed on 360k pages and the last save at 93% is now corrupt. Even though we increased the ram allocation to 8000mb (Xmx8000M) prior to this.
Anyone have any tips as we just lost around 12 hrs worth of info/crawling.
Thank you
After using Xenu for years (and loving it), I recently started using IIS Toolkit and more recently came across Screaming Frog. From a usability perspective I don't think you can go past IIS toolkit, but I must admit I'm really impressed with Screaming Frog (even the free version). Having said that, Xenu will remain on my list of tools to use, and I've even found myself running two or more of them when conducting site audits, just as a sanity check. It's interesting that from time to time I do get different results from each tool. In particular I've seen IIS Toolkit report broken links that are actually fine.
The latest version of Xenu seems to have some bugs, I suggest you download and install the previous https://www.w2o.it/wp-content/uploads/2011/03/XENU-1.3.7.zip Bye Don
Re 'and the winner is?'. Totally correct. Both are excellent bits of kit and I use both regularly and often.
It's a pity Screaming Frog has been getting so much publicity lately. I was hoping that was one the industry was going to keep to itself! ;-)
I love the conclusion… as I never used the SEO frog and Xenu… I didn’t actually dig much in to it so this post is very supportive for the insight of the software.
I will defiantly dig more in to Xenu as it is free and fast and these are the two things a successful software need.
Thank you for sharing the comparison sir!
I have a problem with both when trying to crawl really big sites, 1 million+ pages, when is not possible to do it by sections or folders. I get a memory exhaustion error or similar. Does anyone know and alternative to do it?
No luck trying to find one and I'm already thinking about programming it by myself, but I'd like not to..
Thanks.
I saw today a 64-bit version of Xenu which should be able to (in theory) address more memory.
9. What is the maximum number of URLs that can be checked?There is no fixed number, but it seems to be above one million. The problem is that Windows XP applications have a size of 2GB max.
A 64 bit beta version is available which may or may not allow more URLs. It is based on Microsoft Visual Studio 2010. (Rename the xenu.exe that you already have installed). The 64 bit EXE file is much bigger than the 32 bit version. I suspect that I am making some config mistake with Visual C++ 2010 and that it is packing a ton of library stuff into the EXE file that isn't needed.
Thank you, I'm going to try that.
Are you able to use the IIS SEO Toolkit? It scales well. Totally understand your issue though.
You can increase the memory limit in the SEO spider. It comes with a standard set 512mb allocated of memory (chip rather than disk).
I haven't included it in the FAQ yet, but will explain how to do this, which will help you crawl much, much larger sites.
That said, a million URI is a lot (for a desktop program) - We are working on a few things behind the scenes to make crawling larger sites scale better and also easier to specify the crawl where the IA or structure is not particulary intuitive.
Cheers.
OK thanks. Would be nice to be able to do it with Screamingfrog in a more scalable way, but thinking about how, I see the difficulty on a dektop app.. if you can solve this thing would be really great.
Ahh.. Wonderful! Let's see if Screaming Frog brings a Virus Scanner Warning as well when you try to install (that's what kept me from installing it at work - admins really freak out over some misguided virus alert... *sigh*)
I didn't knew Screaming Frog neither... I find it a very good alternative for Xenu (so I don't use Xenu daily). The platform is very clearly arranged. But as you wrote, Xeno wasn't originally programmed to use it for SEO purpose...
for the records: Xenu runs pretty well on Linux using Wine (though it crashes under certains circumstances :P ). going to try Screaming Frog now...
update: tried to run Screaming Frog on Wine, but unfortunately it doesn't seem to work. since it uses java, they might make a version for linux anyway...
Great write up. Have used Xenu for some time and find myself using it more and more often with new projects -- also relying on it earlier in the process of diagnosing issues.
nice post Dr Pete. Tools like this need a little more love in blogs than they get. I picked up Screaming Frog after Dave Naylor mentioned it a few times at the recent ThinkVis conf over here in the UK. Used it on every site I've touched since.
Both of these softwares are the best in the industry..
Nice post ! But in all honesty both softwares have their own quality. Like you mentioned Frog provides great insight info such as canonical tags where as Xenu doesn't
I am in love with both of these software but I do want to add a negative point of Frog which is that if you are running a big ecommerce website it tends to take longer that my grand mother's shower! It's not easy waiting in the third World country for the bathroom especially when the bloody water runs out!
LOL, like the analogy.
Slow down on larger crawls is often due to reaching the 512mb standard memory limit. Will include how to increase the allowance in our FAQ.
Cheers.
Xenu is great. The functionality is great, but interface is a little bit better than a Unix Bash Shell. Regardless, it's incredibly helpful and powerful, just not the friendliest tool to use.
I`m actually quite satisfied with running Xenu Link on my Mac via Wine / Winebottler. Runs smooth!
Good comparison. I've been using Xenu for a few years now and I still prefer it over Screaming Frog for the sites I do work on (most sites I work on have less than 50 pages on them with exception to a few 1k+ page sites). Overall, the cost of Screaming Frog is actually one of the lower ranking reasons as to why I haven't purchased it. I find that Xenu gets me enough information and a few free web-based applications fill in the rest of the features.
It's nice to have a tool that can knock out several other tools, but sometimes having 2-3 extra bookmarks seems like the more sensible option as oppose to purchasing more software
Dr. Pete thanks for the informative post. I started using Xenu recently but will have to check out the Screaming Frog product. It's nice to see their team engaged in the discussions here. Sounds like someone needs to do a follow up Screaming Frog "Tips + Tricks" post.
I prefer Screaming Frog, since it gives you details about the canonical attributes, Header tags, meta refreshes, and everything you need to assist you in site audits.
I asked a while ago, and they say that there is an option to switch the User Agent coming in the future, because a site scan from Screaming Frog UA may be different than a site scan from Googlebot.
Xenu is a great free Windows option but the interface is nearly unbearable. For our Macs here (The Blue Cube), we use some software called Integrity which was mentioned in another, earlier comment. Whilst it can sometimes be a little buggy, it's extremely useful and has some great options. It's at its most powerful when custom-configured to the site it's crawling.