It’s almost certain that you’ve encountered AngularJS on the web somewhere, even if you weren’t aware of it at the time. Here’s a list of just a few sites using Angular:
- Upwork.com
- Freelancer.com
- Udemy.com
- Youtube.com
Any of those look familiar? If so, it’s because AngularJS is taking over the Internet. There’s a good reason for that: Angular- and other React-style frameworks make for a better user and developer experience on a site. For background, AngularJS and ReactJS are part of a web design movement called single-page applications, or SPAs. While a traditional website loads each individual page as the user navigates the site, including calls to the server and cache, loading resources, and rendering the page, SPAs cut out much of the back-end activity by loading the entire site when a user first lands on a page. Instead of loading a new page each time you click on a link, the site dynamically updates a single HTML page as the user interacts with the site.
Image c/o Microsoft
Why is this movement taking over the Internet? With SPAs, users are treated to a screaming fast site through which they can navigate almost instantaneously, while developers have a template that allows them to customize, test, and optimize pages seamlessly and efficiently. AngularJS and ReactJS use advanced Javascript templates to render the site, which means the HTML/CSS page speed overhead is almost nothing. All site activity runs behind the scenes, out of view of the user.
Unfortunately, anyone who’s tried performing SEO on an Angular or React site knows that the site activity is hidden from more than just site visitors: it’s also hidden from web crawlers. Crawlers like Googlebot rely heavily on HTML/CSS data to render and interpret the content on a site. When that HTML content is hidden behind website scripts, crawlers have no website content to index and serve in search results.
Of course, Google claims they can crawl Javascript (and SEOs have tested and supported this claim), but even if that is true, Googlebot still struggles to crawl sites built on a SPA framework. One of the first issues we encountered when a client first approached us with an Angular site was that nothing beyond the homepage was appearing in the SERPs. ScreamingFrog crawls uncovered the homepage and a handful of other Javascript resources, and that was it.
Another common issue is recording Google Analytics data. Think about it: Analytics data is tracked by recording pageviews every time a user navigates to a page. How can you track site analytics when there’s no HTML response to trigger a pageview?
After working with several clients on their SPA websites, we’ve developed a process for performing SEO on those sites. By using this process, we’ve not only enabled SPA sites to be indexed by search engines, but even to rank on the first page for keywords.
5-step solution to SEO for AngularJS
- Make a list of all pages on the site
- Install Prerender
- “Fetch as Google”
- Configure Analytics
- Recrawl the site
1) Make a list of all pages on your site
If this sounds like a long and tedious process, that’s because it definitely can be. For some sites, this will be as easy as exporting the XML sitemap for the site. For other sites, especially those with hundreds or thousands of pages, creating a comprehensive list of all the pages on the site can take hours or days. However, I cannot emphasize enough how helpful this step has been for us. Having an index of all pages on the site gives you a guide to reference and consult as you work on getting your site indexed. It’s almost impossible to predict every issue that you’re going to encounter with an SPA, and if you don’t have an all-inclusive list of content to reference throughout your SEO optimization, it’s highly likely you’ll leave some part of the site un-indexed by search engines inadvertently.
One solution that might enable you to streamline this process is to divide content into directories instead of individual pages. For example, if you know that you have a list of storeroom pages, include your /storeroom/ directory and make a note of how many pages that includes. Or if you have an e-commerce site, make a note of how many products you have in each shopping category and compile your list that way (though if you have an e-commerce site, I hope for your own sake you have a master list of products somewhere). Regardless of what you do to make this step less time-consuming, make sure you have a full list before continuing to step 2.
2) Install Prerender
Prerender is going to be your best friend when performing SEO for SPAs. Prerender is a service that will render your website in a virtual browser, then serve the static HTML content to web crawlers. From an SEO standpoint, this is as good of a solution as you can hope for: users still get the fast, dynamic SPA experience while search engine crawlers can identify indexable content for search results.
Prerender’s pricing varies based on the size of your site and the freshness of the cache served to Google. Smaller sites (up to 250 pages) can use Prerender for free, while larger sites (or sites that update constantly) may need to pay as much as $200+/month. However, having an indexable version of your site that enables you to attract customers through organic search is invaluable. This is where that list you compiled in step 1 comes into play: if you can prioritize what sections of your site need to be served to search engines, or with what frequency, you may be able to save a little bit of money each month while still achieving SEO progress.
3) "Fetch as Google"
Within Google Search Console is an incredibly useful feature called “Fetch as Google.” “Fetch as Google” allows you to enter a URL from your site and fetch it as Googlebot would during a crawl. “Fetch” returns the HTTP response from the page, which includes a full download of the page source code as Googlebot sees it. “Fetch and Render” will return the HTTP response and will also provide a screenshot of the page as Googlebot saw it and as a site visitor would see it.
This has powerful applications for AngularJS sites. Even with Prerender installed, you may find that Google is still only partially displaying your website, or it may be omitting key features of your site that are helpful to users. Plugging the URL into “Fetch as Google” will let you review how your site appears to search engines and what further steps you may need to take to optimize your keyword rankings. Additionally, after requesting a “Fetch” or “Fetch and Render,” you have the option to “Request Indexing” for that page, which can be handy catalyst for getting your site to appear in search results.
4) Configure Google Analytics (or Google Tag Manager)
As I mentioned above, SPAs can have serious trouble with recording Google Analytics data since they don’t track pageviews the way a standard website does. Instead of the traditional Google Analytics tracking code, you’ll need to install Analytics through some kind of alternative method.
One method that works well is to use the Angulartics plugin. Angulartics replaces standard pageview events with virtual pageview tracking, which tracks the entire user navigation across your application. Since SPAs dynamically load HTML content, these virtual pageviews are recorded based on user interactions with the site, which ultimately tracks the same user behavior as you would through traditional Analytics. Other people have found success using Google Tag Manager “History Change” triggers or other innovative methods, which are perfectly acceptable implementations. As long as your Google Analytics tracking records user interactions instead of conventional pageviews, your Analytics configuration should suffice.
5) Recrawl the site
After working through steps 1–4, you’re going to want to crawl the site yourself to find those errors that not even Googlebot was anticipating. One issue we discovered early with a client was that after installing Prerender, our crawlers were still running into a spider trap:
As you can probably tell, there were not actually 150,000 pages on that particular site. Our crawlers just found a recursive loop that kept generating longer and longer URL strings for the site content. This is something we would not have found in Google Search Console or Analytics. SPAs are notorious for causing tedious, inexplicable issues that you’ll only uncover by crawling the site yourself. Even if you follow the steps above and take as many precautions as possible, I can still almost guarantee you will come across a unique issue that can only be diagnosed through a crawl.
If you’ve come across any of these unique issues, let me know in the comments! I’d love to hear what other issues people have encountered with SPAs.
Results
As I mentioned earlier in the article, the process outlined above has enabled us to not only get client sites indexed, but even to get those sites ranking on first page for various keywords. Here’s an example of the keyword progress we made for one client with an AngularJS site:
Also, the organic traffic growth for that client over the course of seven months:
All of this goes to show that although SEO for SPAs can be tedious, laborious, and troublesome, it is not impossible. Follow the steps above, and you can have SEO success with your single-page app website.
Hi JR.
It's good seeing your happy face someplace other than the Go Fish Digital Slack Channel. Thank you for sharing this very helpful post..
Thanks Bill! Always nice to hear your supportive thoughts wherever they are. :)
One suggestion. For large sites Prerender is expensive. Same happens if you make cache less than a week.
That's why you can built same in-house prerendering solution using PhantomJS or SlimerJS. Of course this will involve some coding and managing infrastructure.
Hey Peter! Could you share any of those in-house solutions? I'd love to take a look at them.
Sure, they're actual some *nix Bash scripting and PHP codes. But saves the day and can be scaled up easy.
Ouch - I used a little bit of PhantomJS on a Nodejs webapp it was a nightmare to work with, kudo's to you for making it work.
I agree with this. The article says, "...while larger sites (or sites that update constantly) may need to pay as much as $200+/month." For enterprise sites, Prerender can cost more than $3,000/month. It's an incredible tool and an amazing team, but wanted to set context for SEOs working on larger sites.
Hi JR,
Thanks for the article. It provides a simple explanation for the basics of AngularJS SEO which, I think, is not understood properly by most SEOs yet.
My only comment would be that if we were wanting to give people a guide on best practices/solutions for AngularJS SEO, surely we must to recommend Isomorphic JavaScript (server rendered JavaScript) over something like Prerender.
But if for some reason you can't implement server rendering, you should definitely use prerender - I just don't think people should jump straight to prerender as the solution in 2017.
Will Critchlow's article from earlier this week mentioned server rendering as "best practice" and I also believe it has to be best practice for any good JS developer.
This article by Bartosz Góralewicz on JavaScript SEO was great, and he says:
"Not doing so (server rendered JS) is simply poor development. It is only OK to use client rendered Angular for content that isn’t publicly accessible (not accessible for Googlebot). For example, your website’s CMS panel, etc."
Both articles mention this video which does a great job at explaining the benefits of server rendering: https://youtu.be/0wvZ7gakqV4
Would love to get your thoughts on this.
Cheers,
David
Hi David! Thanks for chiming in, I think that's a really valid point. We haven't worked with Isomorphic JS, so I really don't have a whole lot of insight to offer there. Based on your explanation though, that sounds like the best framework to use if you can when first constructing your website/app.
Prerender is a nice solution because a lot of times as marketers, we're brought in to help clients after their site's already been constructed with a client-side rendering framework like Angular or React. In cases like that, it's generally not feasible to try and reconstruct the whole framework for an Isomorphic setup. Prerender requires much less development time and resources post-website construction.
So to respond to your comment, yes, absolutely if possible set up some kind of SSR framework. If the website is already built and that's not an option, then our guide above is your best solution.
AngularJS is hot right now so I'm sure we'll only see it become more and more ubiquitous as time goes on. And this is a very real problem. Thanks for helping solve it JR!
Thanks Daniel! Glad this helps!
Great Post!
My experience has been to try your best to keep the 'website' seperate from the SPA. IE have the SPA live behind login/signup nav. It's a lot easier to manage all your SEO and Marketing tasks using a more traditional website platform.
www.mysite.com and app.mysite.com
At first I had the website build into my code and it was a nightmare to 1stly maintain, and 2ndly do all the usual marketing tweaks.
This is a great point. One of the clients we worked with did this exact same thing, where they moved the SPA app portion behind a login portal and set up a standard website for the actual ranking material. This was even after we'd worked to optimize their site through Prerender and using the steps above. It was just too much of a hassle for them to maintain the SPA.
JR, great post here! Thanks for sharing your process and results. I have a couple of follow up questions.
Have you worked with any clients using Isomorphic JS instead of a pre-rendering service? If so, how does that compare results-wise with a pre-rendering service?
Have you tried Brombone and, if so, have you noticed any differences between that and Prerender.io?
Do you know if there is anything people can do to speed up that first page-load, during which the single page app is loaded?
Again, thanks for sharing! Bookmarked.
Hi Everett!
Thanks for the kind words! To answer your first two questions, we haven't worked with either Isomorphic JS or Brombone. If you have any experience with either framework/platform, I'd love to get your insights, those both sound really fascinating.
For the first page-load, some of the best advice is to load as many scripts as possible either asynchronously or inline for the page. This is obviously not a perfect solution, especially since so much of an SPA framework does need to be loaded during the initial call to the server, but any scripts or tags that can be rendered after the initial page load go a long way toward improving PageSpeed scores.
Hope this helps!
Thank you for this post! There isn't a lot of information on AngularJS and SEO yet and all tips, tricks, and advice in this area is extremely valuable!
Glad you like it! I'd love to hear about your experience with Angular, let me know what kind of issues you run into.
Thank you for this blog post it is really useful. Angular JS looks like it could increase user ability on a site by a lot and this is something Google loves to ensure! Cheers for this.
Thanks Cory! Angular is really beneficial for user experience, as long as it doesn't hamper SEO efforts. Hope this helps!
Hey JR,
Thanks for the mention in the post!
Just wanted to clarify, you can crawl AngularJS (or any other JavaScript frameworks) using 'JavaScript rendering' mode in the Screaming Frog SEO Spider.
This will render the web page like a browser, and see the DOM, in the same way that Google do.
More here in a guide I wrote - How To Crawl JavaScript Websites
Cheers.
Dan
Thanks for passing that along Dan! That's super helpful!
Great post JR!
Creating a list of all pages is a great way and I definitely agree with you that fine-tuning of sitemap is an essential factor. But I have a few questions:
First - using Prerender is a risk for cloaking penalty?
Second - Is it true that Prerender website fails in the Google page speed test?
Third - what's your opinion about hash-banging URL format for better page indexing? It converts '#' into 'escaped fragment protocol'.
Fourth - Is it possible to use SSR (server-side-rendering) with RequireJS?
Thank you.
Hi Kuldeep! Great questions, let me try to answer them one by one here:
Hi JR, Would you run a risk of cloaking penalties if the version served to users varies from that served to bots (it's missing formatting, images, various widgets)? Have you ever come across research in this area and how it affects SEO?
If that's the case, there's a chance that could risk cloaking penalties. It depends on how different the versions are. That's where "Fetch as Google" can be really helpful, since you can see the differences in the version served to crawlers compared to the version served to users. In general, the fewer differences between the two, the better.
Google has actually depreciated that AJAX Crawling Scheme you linked to back in 2015 (https://webmasters.googleblog.com/2015/10/deprecat...) and it's not really recommended. A client of ours initially had !# in their urls when we started working with them and the indexation rate wasn't pretty. It improved when we moved away from !# and started using normal urls instead.
Hello JR Ridley,
Thanks for this article, it's good to have a validation of this technique.
I'm actually working with a big client in France that have the same issue and i recommand exactly the same actions.
We will see in few months the results of that.
Glad to hear it Matthieu! Let me know how everything goes with your client!
There still many webs and aps which are not using angular js, because of many issues. This post might help those.
I certainly hope so! Thanks for the support Richa!
Nice, another article about JS. I should start learning, really. I see that it's going to be very important for SEOs to know JS in the future. I can read the code but not write... Thanks for sharing!
No problem! Glad this helps Martin!
Hello! Great article, thank you! How do you deal with this recursive loop that keeps longer and longer URL strings for the site content? Do you have any problem with titles and snippets in Search Results?
Hi Kate! Great question. The recursive loop was an issue with a dynamically generated canonical tag, but since the site was built on Javascript, that just made the issue repeat everytime the content was reloaded by a crawler. And we've definitely had trouble with titles and snippets, Google has a lot of trouble recognizing meta content for SPAs. Although it's not a perfect fix, Prerender was the best solution we found here.
Hope that helps!
Hey JR Ridley,
Nice to meet you. No doubt excellent post indeed!
Last week I was reading about AngularJS in WPSchools about this JavaScript framework.
It can be added to an HTML page with a <script> tag and extends HTML attributes with Directives, and binds data to HTML with Expressions.
Anyway, you did a great job mentioned traditional page life cycle and the whole single page applications.
Nice work. JR keep it up!
Hey Shakoat! Is that for applying the framework to a single page or to an entire website application? I know YouTube uses a format that applies the framework to a specific page without applying to the whole site.
Thanks for the kind words! :)
I agree with all the points in the article but having the experience of working on a SPA website with approx. half a million pages and using custom build pre-render, it is always a challenge to make sure Google bot can read it properly.
We launched this SPA website 2 years ago and ranking good for approx 263K keywords as per SEMRush and gets about 700K monthly organic visits.
However, Google bot is still at odds where they drop some of our pages on daily basis after they can't see the HTML snapshot page. Our cache is hosted on AWS for up to 4 days which could be an issue but it is hard to troubleshoot or it could be G bot not waiting enough to see the snapshot page. This is something we are trying to isolate but it is work in progress.
The other challenge is to show 404 errors as they are client side and therefore GSC only reports very few 404 but 5xx errors are quite frequent.
John Mueller from Google did a very good YT on this which is worth watching https://www.youtube.com/watch?v=JlP5rBynK3E
Thanks for adding in your thoughts! Have you had any success adjusting how often your pages are cached and refreshed?
Fascinating. I always heard (and believed) that linked YouTube content would automatically be a boost for Google ranking since they want to boost their products. I would have assumed that they built their own service to handle the SPA handshake. Great information.
Good post, JR Ridley!
We've deployed a prerender solution a couple of months ago for our AngularJS site (although it's only our homepage that's built with AngularJS) and it seems to be doing fine (no drops), however, I have this feeling (have no data on this) that Google doesn't like the prerendering and its stagnating our organic results.
What's your view on prerendering? Besides the fact that it is probably the best solution for any AngularJS site, but do you think, the fact that Google deprecated the AJAX crawling scheme, prerendering is causing negative results regarding SEO?
Hi Gert! Prerendering is definitely an imperfect solution, but it really is the only viable solution we've found for SPAs. It's likely your prerendered homepage faces some additional hurdles in organic search that other traditional websites don't, such as updated content, user visibility, etc. I don't think Prerender would harm your SEO efforts at all, but you still may face other obstacles that non-SPA websites do not.
Best of luck! I'd love to hear any results you get!
Thank you for sharing your knowledge JR!
I'll keep this in mind for my project
Hi JR, Thanks for the post.
I was wondering, do you happen to know which angularjs version does youtube.com has? The reason I'm asking is that in version greater than angularjs1.x you can use https://universal.angular.io/. (I'm not familiar with sites that already have that in production)
Another issue that I would like to raise:
Using the current paradigm with prerender and _escaped_fragment_ requires that google crawls twice each page, which consume a valuable crawl time.
Do you know if serving the cached pages from prerender on first page hit, by detecting the user-agent will be considered as cloaking?
i.e. serving cached page to user-agent: googlebot. otherwise, serve the page from the server.
Basically both pages content are almost the same, beside a javascript content that is different
Thanks
Apparently it is still not viable to create AngularJS webs. There are many difficulties.
Yes! Lots of difficulties! We understand that there's a lot of benefit for users of building an SPA, but it comes with a lot of obstacles on the SEO side.
Is the optimization of images really significant against the name of the images in the seo ?
Hello JD Ridley
Thanks for the Great Article, SEO for AngularJS is really disturbing. Been Considering the Prerender.io Option for a while. I think i am finally taking that route
Glad to hear it! Thanks for the kind words, and let me know how your Prerender implementation goes.
Fantastic article, thank you
Thank you for the kind words!
This is really a nice and informative.Much obliged for making such a cool post which is truly exceptionallywell written.Thanks for sharing it
Thanks Tony! Hope you find it helpful!