I recently published the results of my JavaScript SEO experiment where I checked which JavaScript frameworks are properly crawled and indexed by Google. The results were shocking; it turns out Google has a number of problems when crawling and indexing JavaScript-rich websites.
Google managed to index only a few out of multiple JavaScript frameworks tested. And as I proved, indexing content doesn’t always mean crawling JavaScript-generated links.
This got me thinking. If Google is having problems with JavaScript crawling and indexing, how are Google’s smaller competitors dealing with this problem? Is JavaScript going to lead you to full de-indexing in most search engines?
If you decide to deploy a client-rendered website (meaning a browser or Googlebot needs to process the JavaScript before seeing the HTML), you're not only risking problems with your Google rankings — you may completely kill your chances at ranking in all the other search engines out there.
Google + JavaScript SEO experiment
To see how search engines other than Google deal with JavaScript crawling and indexing, we used our experiment website, http:/jsseo.expert, to check how Googlebot crawls and indexes JavaScript (and JavaScript frameworks’) generated content.
The experiment was quite simple: https://jsseo.expert has subpages with content parsed by different JavaScript frameworks. If you disable JavaScript, the content isn’t visible — i.e. if you go to https://jsseo.expert/angular2/, all the content within the red box is generated by Angular 2. If the content isn’t indexed in Yahoo, for example, we know that Yahoo’s indexer didn’t process the JavaScript.
Here are the results:
As you can see, Google and Ask are the only search engines to properly index JavaScript-generated content. Bing, Yahoo, AOL, DuckDuckGo, and Yandex are completely JavaScript-blind and won’t see your content if it isn’t HTML.
The next step: Can other search engines index JavaScript?
Most SEOs only cover JavaScript crawling and indexing issues when talking about Google. As you can see, the problem is much more complex. When you launch a client-rendered JavaScript-rich website (JavaScript is processed by the browser/crawler to “build” HTML), you can be 100% sure that it’s only going to be indexed and ranked in Google and Ask. Unfortunately, Google and Ask cover only ~64% of the whole search engine market, according to statista.com.
This means that your new, shiny, JavaScript-rich website can cost you ~36% of your website’s visibility on all search engines.
Let’s start with Yahoo, Bing, and AOL, which are responsible for 35% of search queries in the US.
Yahoo, Bing, and AOL
Even though Yahoo and AOL were here long before Google, they’ve obviously fallen behind its powerful algorithm and don’t invest in crawling and indexing as much as Google. One reason is likely the relatively high cost of crawling and indexing the web compared to the popularity of the website.
Google can freely invest millions of dollars in growing their computing power without worrying as much about return on investment, whereas Bing, AOL, and Ask only have a small percentage of the search market.
However, Microsoft-owned Bing isn't out of the running. Their growth has been quite aggressive over last 8 years:
Unfortunately, we can’t say the same about one of the market pioneers: AOL. Do you remember the days before Google? This video will surely bring back some memories from a simpler time.
If you want to learn more about search engine history, I highly recommend watching Marcus Tandler’s spectacular TEDx talk.
Ask.com
What about Ask.com? How is it possible that Ask, with less than 1% of the market, can invest in crawling and indexing JavaScript? It makes me question if the Ask network is powered by Google’s algorithm and crawlers. It's even more interesting looking at Ask’s aversion towards Google. There were already some speculations about Ask’s relationship with Google after Google Penguin in 2012, but we can now confirm that Ask’s crawling is using Google’s technology.
DuckDuckGo and Yandex
Both DuckDuckGo and Yandex had no problem indexing all the URLs within https://jsseo.expert, but unfortunately, the only content that was indexed properly was the 100% HTML page (https://jsseo.expert/html/).
Baidu
Despite my best efforts, I didn’t manage to index https://jsseo.expert in Baidu.com. It turns out you need a mainland China phone number to do that. I don’t have any previous experience with Baidu, so any and all help with indexing our experimental website would be appreciated. As soon as I succeed, I will update this article with Baidu.com results.
Going beyond the search engines
What if you don’t really care about search engines other than Google? Even if your target market is heavily dominated by Google, JavaScript crawling and indexing is still in an early stage, as my JavaScript SEO experiment documented.
Additionally, even if crawled and indexed properly, there is proof that JavaScript reliance can affect your rankings. Will Critchlow saw a significant traffic improvement after shifting from JavaScript-driven pages to non-JavaScript reliant.
Is there a JavaScript SEO silver bullet?
There is no search engine that can understand and process JavaScript at the level our modern browsers can. Even so, JavaScript isn’t inherently bad for SEO. JavaScript is awesome, but just like SEO, it requires experience and close attention to best practices.
If you want to enjoy all the perks of JavaScript without worrying about problems like Hulu.com’s JavaScript SEO issues, look into isomorphic JavaScript. It allows you to enjoy dynamic and beautiful websites without worrying about SEO.
If you've already developed a client-rendered website and can’t go back to the drawing board, you can always use pre-rendering services or enable server-side rendering. They often aren’t ideal solutions, but can definitely help you solve the JavaScript crawling and indexing problem until you come up with a better solution.
Regardless of the search engine, yet again we come back to testing and experimenting as a core component of technical SEO.
The future of JavaScript SEO
I highly recommend you follow along with how https://jsseo.expert/ is indexed in Google and other search engines. Even if some of the other search engines are a little behind Google, they'll need to improve how they deal with JavaScript-rich websites to meet the exponentially growing demand for what JavaScript frameworks offer, both to developers and end users.
For now, stick to HTML & CSS on your front-end. :)
Good post. Many times we focus so much on Google that we leave the other searchers a bit aside, even though they do not bring as much traffic as any sum and your potential customers can enter through any channel. Why limit ourselves to only one?
Exactly! :) Thanks Fran!
I now try to optimize for several the problem that I find is that the facilities that gives us Google is not found in other search engines. One question I have. Do other search engines use Google to make their positions? For example Yahoo uses Google to show you the results?
Very interesting post. I thought Bing and Yahoo would be better at indexing Javascript frameworks.
Also, what the flip is DuckDuckGo? :)
DuckDuckGo is a new(ish) search engine that doesn't collect user data from searchers (or at least not as much as the other engines). The idea is that it maintains searcher privacy, but provides less personalized search results.
Hi Bartosz, great post! Just to add that you can actually get a Baidu Webmaster Tools account, I've done it successfully. Just use this link which goes to the same page as that mentioned in the link you provide, except it gives you the option of choosing the country calling code.. Unfortunately the tool is Chinese only, but Google Translate will do. After spending some time on it, i successfully indexed a website I managed :)
Thanks AskewSEO!! I will definitely give it a shot!
Quite Interesting post! And It's shocking to know that Ask is using technology parallel to Google but other search engines not who are in existence very earlier than it.
I saw the article about JS and SEO at your website few weeks ago and it was absolutely great. And now MOZ - another checkpoint. Keep it up and don't stop making experiments!
Thank you Radek!
Great piece! I wrote an article a while back about our experience running SEO for AngularJS clients, and we had a lot of the same issues you pointed out with Javascript crawling and indexing. We didn't experiment much with search engines beyond Google, so this information is super helpful for adding another layer of depth to the equation.
Do you have any experience using the ScreamingFrog Javascript rendering configuration (as opposed to the standard AJAX crawling scheme)? Is that effective for crawling these JS sites?
Hi JR, great question, I actually wrote a blog post about this, you can find it here https://goralewicz.com/blog/javascript-seo-how-to-... in general, Screaming Frog seems to be the best option for crawling JS rich websites.
Great post Bartosz, always great to see other search providers picking up momentum. Lets hope they jump on the JavaScript band wagon sooner rather than later.
Thanks Glenn!
I have several problems, there a lot of new technologies that you need to implement and in the other hand you have to be sure that is useful, in my case I I developed a hybrid, it was a website, app and cms, was not a big deal was based on Angular and Database the idea was create the best experience for the user example if we publisha post this post will be served based on the user device, mobile, desktop, tablet or browser. At the end was reject because we had a lot of problem to implement without losing rank.
Roman - maybe you can give it another try with Server Side Rendering?
Haha! Love the AOL commercial!
I have built this tax calculator in pure AngularJs, even the meta titles are generated via JavaScript, and all the content is updated as the script does the calculations after page load. Yet, Google has done an excellent job in indexing the content, including titles and description (all Js generated): https://www.google.co.uk/search?q=site%3Aincome-ta... . Not only it has indexed it, but I've had no difficulties ranking on first page for longer tail, lower volume keywords (so far, it's been about 2 weeks since I've finalised the tax calculator).
However, Bing is acting like a drunk guy in reading the website: https://www.bing.com/search?q=site%3aincome-tax.co.... - it can't even read the page titles. Looking forward to seeing the evolution of the rankings (mostly on Google), if any of you guys want to see this and study how the rankings of a JavaScript generated website will evolve, I'm even sharing my rankings report that's updated every 24 hours: https://ranksreports.com/RK8U72QMGK2B8PBR108M0LUI8VFK39
I hope the future of crawlers will be a perfect browser imitation, as it would open up the doors to unlimited interactive web applications that not only are useful, but they could also get some organic traffic.
Thank you for your comment Ferencz, in our experiments, we found that Angular 1 is indeed indexable, but it is really difficult to rank and to get a decent indexation/crawling budget with Angular websites. What is your experience?
Bartosz, I'm watching the rankings day by day, what I've noticed is that if I keep the link veocity high (i.e. 3-4 new genuine backlinks/week), then it goes up, and then if I stop with getting any new links for a few days, it starts moving down. The competition is high though, and I gues I just have to make the website even more awesome, make it 100% mobile friendly, and I'm very confident that it will get on the first page for "income tax calculator" within 1 year.
Awesome post Bartosz, Love it !
It helps me to take into account a few new things that I did not know in my SEO strategy.
Thank you Mike!
Good stuff. One nitpick: It's 'indexing', not 'indexation'. The latter is a term exclusive to the finance industry. It is not used in information retrieval contexts.
Thanks Barry, I was actually looking forward to your feedback! I love your articles about crawling and indexing.If you only have one nitpick from reading this article, it means I did something right :) I will make sure this is corrected in the article. THanks again!
Thanks for the learning moment, Barry! We've got that all fixed :)
I was quite impressed when I read your first study about Javascript indexing, and yet you come again with another interesting one again. Thank you for doing these type of research and publishing it to the world. I'll keep that for future references if I need to.
Thanks Angel! I was surprised to see how well my experiments were received. I will definitely make sure to keep you guys up to date with more of our findings as JS crawling topic is extremely complex and exciting.
Great post Bartosz!
It helps me to take into account a few new things that I did not know in my SEO strategy.
Thanks Javier!
I would never have said that Ask was able to index so well generated content with JaviScript
Interseting post!!
I agree with the assessment that search engines need to jump on the javascript bandwagon. I included this in my roundup post, and my main takeaway was: Use HTML and CSS for now, but search engines need to adapt to Javascript-rich websites. It's time.
I wonder if anyone is still using Ask, DuckDuckGo or Aol and optimizing for it. But great, that you included even those into your study here.
My education website is completely on javascript, when I disable javascript, the site doesn't load fully. But my site is getting 300,000+ page views monthly 70% from Google. It means Google crawls it perfectly, I don't know which version is the javascript being used on my site. Should I use a theme that is not javascript based? Please if anybody can explain!
Hi Fahad, can you share your website's URL?
Good Insight. On a related topic......If the search engines can index dynamically created content how will they provide a direct link to that piece of content within the search results?
Good question Eddie, Best practices mention (SEO and development best practices) mention that you need to match URL to content wherever possible. This is not only and SEO, but also UX problem.
Well researched article, This was what I looking for past few days but couldn't find any satisfactory response.
Thank you Goyalaakash!
Fascinating piece.
How does MOZ do at indexing javascript? Does MOZ DA/PA reflect proper Google-esq javascript indexing or would it be closer to DuckDuckGo?
This is a good question, but unfortunately I can't answer it. I would guess that MOZ doesn't crawl JS. Most (if not all) crawlers don't do that. There is an interesting quote from Ahrefs CEO https://searchengineland.com/know-google-sees-javas... "To execute JS for every page at our scale [would] require 10,000-15,000 servers, and we believe our customers are not ready to pay for that yet" and I think this answers your question. I think it is still too expensive to crawl JS on a mass scale.
Thanks for the ping Bartosz, aHrefs is executing now https://ahrefs.com/blog/crawling-javascript/, and from what I understand, Moz isn't prioritizing JS crawling at this time.
Very interesting study. Just reinforces the best SEO practices that it is best to have HTML pages and text links if you want everything indexed. Having links in Javascript may not get the linked page indexed. Similarly using Angular and having an infinite scroll single page website may result in a very small portion getting indexed, if at all.
HTML is definitely the safest option. At the same time, JS is not in inherently wrong. You can use isomorphic JS as I wrote in https://www.searchenginejournal.com/javascript-seo... at the same time, you need to remember, that using JS needs MUCH more expertise thank HTML/CSS. For most webmasters, sticking to HTML is the safest and the cheapest/fastest way to develop an SEO friendly website.
excellent article, I always had my doubts about the seo in javascript, It has helped me a lot to clarify some points.
Thanks!
Thanks Miracles!
What about Python programming? Why did you leave python out?
Hi Kris, Python is only used for backend programming, there is no such thing (to my best knowledge) as front end Python and even if there was, browsers only support HTML, CSS and JS.
This is a little odd. Yahoo uses Bing to do most of its search (along with some Google traffic), DDG uses Yahoo on top of Bing. Yandex is independent. Ask apparently does some aggregation too. Point being that what you really want to be testing is the people who are actually doing the web crwaling. You could save some time that way.
Probably the easiest way to look at what exists is to look at what web crawlers exist, i cannot add links (they are set to no follow), but you can do a quick search for what web crawlers exist.
I was thinking about this approach, but it was much easier to simply submit a website to different browser and see if it is indexed. Going deeper into specific crawlers is definitely an interesting topic though.