Read about the 5 new upgrades we've added to Site Crawl (as of September 2017) here!
First, the good news — as of today, all Moz Pro customers have access to the new version of Site Crawl, our entirely rebuilt deep site crawler and technical SEO auditing platform. The bad news? There isn't any. It's bigger, better, faster, and you won't pay an extra dime for it.
A moment of humility, though — if you've used our existing site crawl, you know it hasn't always lived up to your expectations. Truth is, it hasn't lived up to ours, either. Over a year ago, we set out to rebuild the back end crawler, but we realized quickly that what we wanted was an entirely re-imagined crawler, front and back, with the best features we could offer. Today, we launch the first version of that new crawler.
Code name: Aardwolf
The back end is entirely new. Our completely rebuilt "Aardwolf" engine crawls twice as fast, while digging much deeper. For larger accounts, it can support up to ten parallel crawlers, for actual speeds of up to 20X the old crawler. Aardwolf also fully supports SNI sites (including Cloudflare), correcting a major shortcoming of our old crawler.
View/search *all* URLs
One major limitation of our old crawler is that you could only see pages with known issues. Click on "All Crawled Pages" in the new crawler, and you'll be brought to a list of every URL we crawled on your site during the last crawl cycle:
You can sort this list by status code, total issues, Page Authority (PA), or crawl depth. You can also filter by URL, status codes, or whether or not the page has known issues. For example, let's say I just wanted to see all of the pages crawled for Moz.com in the "/blog" directory...
I just click the [+], select "URL," enter "/blog," and I'm on my way.
Do you prefer to slice and dice the data on your own? You can export your entire crawl to CSV, with additional data including per-page fetch times and redirect targets.
Recrawl your site immediately
Sometimes, you just can't wait a week for a new crawl. Maybe you relaunched your site or made major changes, and you have to know quickly if those changes are working. No problem, just click "Recrawl my site" from the top of any page in the Site Crawl section, and you'll be on your way...
Starting at our Medium tier, you’ll get 10 recrawls per month, in addition to your automatic weekly crawls. When the stakes are high or you're under tight deadlines for client reviews, we understand that waiting just isn't an option. Recrawl allows you to verify that your fixes were successful and refresh your crawl report.
Ignore individual issues
As many customers have reminded us over the years, technical SEO is not a one-sized-fits-all task, and what's critical for one site is barely a nuisance for another. For example, let's say I don't care about a handful of overly dynamic URLs (for many sites, it's a minor issue). With the new Site Crawl, I can just select those issues and then "Ignore" them (see the green arrow for location):
If you make a mistake, no worries — you can manage and restore ignored issues. We'll also keep tracking any new issues that pop up over time. Just because you don't care about something today doesn't mean you won't need to know about it a month from now.
Fix duplicate content
Under "Content Issues," we've launched an entirely new duplicate content detection engine and a better, cleaner UI for navigating that content. Duplicate content is now automatically clustered, and we do our best to consistently detect the "parent" page. Here's a sample from Moz.com:
You can view duplicates by the total number of affected pages, PA, and crawl depth, and you can filter by URL. Click on the arrow (far-right column) for all of the pages in the cluster (shown in the screenshot). Click anywhere in the current table row to get a full profile, including the source page we found that link on.
Prioritize quickly & tactically
Prioritizing technical SEO problems requires deep knowledge of a site. In the past, in the interest of simplicity, I fear that we've misled some of you. We attempted to give every issue a set priority (high, medium, or low), when the difficult reality is that what's a major problem on one site may be deliberate and useful on another.
With the new Site Crawl, we decided to categorize crawl issues tactically, using five buckets:
- Critical Crawler Issues
- Crawler Warnings
- Redirect Issues
- Metadata Issues
- Content Issues
Hopefully, you can already guess what some of these contain. Critical Crawler Issues still reflect issues that matter first to most sites, such as 5XX errors and redirects to 404s. Crawler Warnings represent issues that might be very important for some sites, but require more context, such as meta NOINDEX.
Prioritization often depends on scope, too. All else being equal, one 500 error may be more important than one duplicate page, but 10,000 duplicate pages is a different matter. Go to the bottom of the Site Crawl Overview Page, and we've attempted to balance priority and scope to target your top three issues to fix:
Moving forward, we're going to be launching more intelligent prioritization, including grouping issues by folder and adding data visualization of your known issues. Prioritization is a difficult task and one we haven't helped you do as well as we could. We're going to do our best to change that.
Dive in & tell us what you think!
All existing customers should have access to the new Site Crawl as of earlier this morning. Even better, we've been crawling existing campaigns with the Aardwolf engine for a couple of weeks, so you'll have history available from day one!
Find & fix your site issues now
Stay tuned for a blog post tomorrow on effectively prioritizing Site Crawl issues, and for a more in-depth look into how to use Site Crawl, check out the recorded webinar.
I've got to pay attention more in meetings, this is even better than what I had thought! AWESOME!
Amazing stuff and major kudos to Dr. Pete and the Moz team for relaunching the new site crawl. I'll be 100% honest--for the past year as I grew in my knowledge of SEO, I realized more and more how the Moz site crawl wasn't really helpful. (I switched over to SEMrush and Screaming Frog last year and have been using them for technical SEO crawls for the majority of my projects.) But with this update it feels like I will be able to confidently come back to Moz for all my SEO crawling needs. (YAY!)
Thank you again and looking forward to playing with this a lot over the next week!
What's the limit on the number of pages that can be crawled?
Sorry, we were still tweaking a couple of details as the post was being written, but the pricing page has all current limits. All customers will see their complete list of crawled pages now, whereas before we only displayed pages with issues, so hopefully everyone is getting more useful data.
Existing customers will get the higher of their current limits or the new limits, so no one will lose pages.
Great tool, have been using it for the last couple of days. It is a lot more useful for me now compared to the older version. I absolutely love it when you click on a URL to see all issues and important information for a specific page.
There are some things however I would love to see added to make my life as a SEO much easier:
- Ignore all button: currently you can only ignore 25 items at a time (ignoring 3.7k too long meta titles for products pages wasn't that fun).
- Negative URL filter: I would love to see a function where you could exclude a part for a URL (for instance excluding products/category pages if the URL looks something like: domain/category/collection/product). In combination with the above "ignore all button" this could be a massive time saver.
- Option to hide ignored items: Let's say I ignored 95% of all the URL's for a specific issue. As a SEO I want to hide the URL's I ignored to only focus on the active ones that I still need to fix.
- Ignored Label in exported CSV: It would be great to see a column in the CSV to quickly filter out all the issues I ignored in Excel.
I know this is just the first iteration of this new tool, but I just wanted to let you guys know. Keep up the good work!
Hello Mark!
Glad to hear it is working well for you!
I would say that this feedback is pretty consistent with other feedback we have had around the 'Ignore Issues' feature - we are going to be discussing ways to address this week - look out for updates soon!
Thanks!
Jon
Thank you for taking the time to write up your thoughts! It means a lot to us!
Sorry, been on vacation, but thanks for the detailed feedback, Mark! We're reviewing all of it ASAP. Definitely realize we need to introduce some bulk ignore options quickly.
The ignore feature is worth its weight in GOLD. Thanks for that enhancement!
WOW, that's all I can say. Great!
I am so excited about this!
is the hreflang included in the analysis? Didn't had time to check, sorr
Not yet -- we're working on some more complex rules for that and canonicals. Gets really tough to tell people if their tags are "right" because intent is involved.
Is Structured Data something that you will be introducing to crawls at some point? I'm thinking it will be growing in importance to Google over the next year or so. Thanks!
Hi Brian - yes this is on our radar. We have got a bit of iteration to do on what we just launched but then we will be looking to some new features. Thanks for the comment and suggestion!
Wow this is awsome! Have been waiting a while for Moz to put out a full crawler like Aardwolf. Cant wait to start using it.
Awesome announcement buddy. Looking forward to poking around. I'm a download to csv and analyze in excel sort of guy - I hope there's lots of cool stuff to find!!
Is it weird that I'm most excited that I can ignore issues? I like to keep things neat and tidy, so this is perfect.
Awesome! Finally I can start using something different than Screaming Frog. Cheers, Martin
good tool!! Everything that helps keep our websites clean and tidy better. The easier we put it to Mr. Google the better.
It's a great news for Moz Pro customer. Now they are easily and fastercheck Crawler Issues..
Has the "Email CSV" option that shows on the front page of the old Site Crawl front page gone away? I've used that for years to get a CSV of all the pages and their associated issues. It would be much more time consuming to go to each report type and download the individual CSVs.
Sorry for any confusion -- you can export the full CSV of all crawled pages from the "All Crawled Pages" section. The new home-page is meant to be more of a summary, so we just moved that function over. It's right above the table (upper-right, below graphs),
I did see the All Crawled Pages CSV download, but as far as I can tell, that report only shows the number of issues per page and not what issues affect each page. On the old site crawl's "Email CSV" option, it showed every page with an issue and a true or false flag for each issue type.
Ah, understood. To allow on-demand CSVs, we opted to split them into an "All Pages" report (with a bit less info for page) and then separate, category-based reports that display each issue. Those latter reports are one line per issue now, to allow easier imports and data manipulation.
There's not currently an all-in-one -- sorry about that. Since this is a major overhaul, though, we will be closely monitoring feedback (including comments on this post) to evaluate next steps. We hope to iterate reasonably quickly this year.
Still having issues getting our site crawled. Were any changes made to your version of python to support SNI?
Yes, the new crawler architecture should fully support SNI -- that was a big priority for us. Could you PM me with the campaign, and I'll have the team take a look?
Hey there! I took a look at your campaign and it looks like we have crawled over 11k pages of the site, so you should be getting much better data now. If you have any other issues or questions with the data you are seeing in the new site crawler, please email [email protected] so that we can get things sorted out for you as soon as possible.
thats awesome that they have made a better version of it! looking foward to using it!
I am also facing the same issues in crawling my website
Hey Surbhi,
I tried to take a look at your campaigns, but it looks like there isn't an active subscription under this account. If you email [email protected] and include the name of your campaign and the login email address for your account, they can look into any issues for you.
Nice guide & tools to find crawling issue of your webpages & fix them quickly.