Earlier this year, jtkaczuk wrote a YOUmoz post about “Using Twitter as a Sitemap”. After reading it I began to think about the power of Twitter and if using Twitter more can help indexation. Many Twitter users will tweet about new post or products on their account hoping to draw attention and links from their followers. What if this process can also help with getting more pages indexed and indexed faster? I was surprised with the results of this quick little experiment that I threw together in a few months.
Experiment Setup
The experiment started with 15 local clients of mine who often tweet about new products or posts on their Twitter accounts. These accounts vary in followers from 75 to about 1500. While I did not have direct control over these accounts, I was able to track when a new product was added, tweeted, crawled by GoogleBot, and indexed by Google via a PHP script I wrote and installed on their CMS. Along with tracking those, I monitored when the number of RTs, when the product was indexed, and if it stayed indexed for at least 48 hours after it was launched.
For each product or post that was launched, they were placed in one of three categories for 48 hours:
- Twitter Links Only – 48 hours with no internal links and 1 tweet only from the orginal account
- Site Architecture Links Only – No tweets about product or post, only internal links and sitemaps generated
- Both Twitter & Site Architecture Links – Both tweets, internal links, and sitemaps to post or product
After the 48 hour observation period was over, the products or posts were launched like a normal, which included tweets, internal links, and anything else my clients might do to promote it. We also stopped collecting data at that point.
Experiment Warning
As Rand and Ben always say, correlation does not imply causation. Nor do I encourage that you SPAM Twitter with a whole bunch of links to content that is not useful to your followers. Take the results of this experiment and try to find where you can fit them in your business without upsetting and losing your followers.
Experiment Data Summary
During the course of the experiment: 120 products or posts where published - 40 in each of the categories above, there over 96 RTs, over 350 GoogleBot visits, and an 87% indexation rate. Here are some quick highlights of the findings:
- Twitter Only Launch
- GoogleBot averaged its first visit within 78 seconds of the first tweet.
- Tweets with more than 3 RTs were indexed 325% faster, along with 125% more of its products and post indexed than tweets with no RTs.
- Average indexation of the post or product was different depending on number of RTs
- 3 or more RTs had an average indexation time of 8 hours, with 86% indexed.
- 1 or no RTs had an average indexation time of 26 hours, with 69% indexed.
- Internal Links Only Launch
- GoogleBot averaged its first visit within 2.5 hours of the first internal link on the site.
- Average indexation of the post or product was 8.5 hours, with 85% indexed.
- Both Internal Links and Twitter Launch
- GoogleBot averaged its first visit within 82 seconds of being launched.
- Average indexation of the post or product was again different depending of the number of RTs the Twitter updated received.
- 3 or more RTs had an average indexation time of 4.25 hours, with 88% indexed.
- 1 or no RTs had an average indexation time of 5 hours, with 79% indexed.
Experiment Raw Data
Experiment Conclusion
The data concludes that creating your new product or post with internal links along with a tweet that gets 3 or more RTs, will help in increasing the time and rate at which they get indexed. While the data may show there is evidence that this technique will help your site increase its indexation and crawl time, I would advise you to do it with caution and care. All of my clients took care not to launch more than 1 product a day and did continue to tweet other things besides the new products launched. My personal warning is to remember that Twitter is designed for your clients and not as a launching pad for Google, it would be horrible to see your account lose its following due to mass product tweeting. What are your feelings or experiences on using Twitter to increase your indexation?
Great article. Did you also test:
I'd also like to know if short URLs have any impact on this study
I didn't think to look at the mT of the twitter accounts. Looking back at them, they are all very similar around 5.22 to 5.9. All tweets where using bit.ly as the shortener.
Interesting..
I wondered about this myself, using shortening tools. I always thought Google won't bother to index it because it's not an actual URL.
Good to know! Interesting post and research too...
Personally I don't think that shortened links are a big problem, being a 301 redirect to the main URL.
Was there any correlation/causation between the number of followers and the crawl/index rates?
(It was a great article by the way. Extremely interesting.)
Hey Nicholas,
I wasn't able to notice any difference between my clients that had more followers. It really only seemed to increase things when the tweet was RTed more than 3 time.
Which seems to make sense. Getting a following is much easier than retweets. Everyone may not need Google's attention after 90 seconds, but this data suggests that retweeting is a strong signal of relevancy. That in itself is worth retweeting ;-)
I agree. However, RTing also has the potential to be exploited.
For this reason, I doubt it will be a major ranking factor. As Matt Cutts has said, popular content doesn't equal relevant content.
Interesting test. Be good to know whether the number of followers may also be a factor(trust). I think RT's would also come into play.
I think for the average site selling products or blog posts etc, a few minutes or a few hours may not make any difference but for news and press release sites it may be more important to have those pages indexed faster.
Plan on posting snippet of that php code?
I'll see what I can come up with for you. It's not exactly the cleanest code I've written in the last few months.
Really great information! I really am liking twitter quite a bit more than Facebook as far as social media for businesses... and now another added bonus.
Way to go Casey. You are without a doubt well on your way to becoming VP of "DataheadsRus" (Ben is the CEO)
Thanks for taking the time to research, write this up and share it in the middle of your house being turned upside down.
I'm nowhere near Ben! He is on another plant when it comes to data! I think we need to see the "real" goodnewscowboy in his avatar for a little while.
Define 'real'. :-p
Mmmm, a picture of a human face....
Show us your Face cowboy!
Very interesting article, will carry out some testing to see how we can utilise this to the benefit of our users.
Cool experiment, thanks for the data. It looks like Google is using Retweets as a sort of trust factor for twitter. I was wary of using Twitter as a sitemap, but it could help in certain instances. I agree with seo-himanshu in that it would be useful for a news site but otherwise it won't matter too much if your content is indexed in 3 hours or 8 hours. It depends on how time-dependent your information is.
It also depends of the strength of your competitors. It could be a real advantage to be indexed quicker in order to get more external links on your fresh content.
Interesting article. I hadn't really thought about the impact of tweets on indexation speed before. Wonder multiple retweets has an impact on rankings (however slight).
Thanks for an interesting post.
I'm curious about the process of tweeting about new products. Was it automated somehow or did the client manually tweet about it after posting a new product?
I client manually tweeted about it and my bot picked it up and started monitoring then.
I wonder how it workd for the relatively new sites vs the established ones. For the sites that have been established with Google and "earned" their super fast indexation rate this may be irrelevant. But it is a very interesting study, thank you!
Thanks for putting this together. I would love to see folks doing more in depth experiments in this department. Certainly more data would help the group understand this better.
Will changing the URL shortner change the indexing speed?
buzoganylaszlo this is already happening! in Australia we don't use twitter much, but plenty of people use Facebook. I find that the status updates is basically simliar but facebook providing more features thus it's the winner
While the warning to avoid spamming about new products and services is definitely good. Also, remember that your' expertise in your' subject is why they follow you in the first place.
This tweet may cause lost follower syndrome:
We are glad to launch ABC today. You guys all bought XYZ, so I'm sure that we will succeed... again.
Yes, it's an exaggeration but many sound like that.
This reflects why you tweet and why your' customers follow you:
@happy_customer we heard you were looking for X and we are pleased to say that we can now provide it. If anyone else is interested, call for a demo. Please share with any interested friends
Always remember, your' followers follow you and it is a license to promote your' services, but do so responsibly and considerately.
Very interesting When I blog I have it on auto to tweet then also on my personal twitter and I have noticed I get indexed very quickly.
Good post, but what my question is.... Why? I get it, we would like to have new pages indexed asap, but does it really make a difference, if they are indexed 45 seconds or a couple of minutes after they are published?
It matters if you run a news site.
Well it really depends on what kind of site you run. If you have a news site like seo-himanshu said, you want that article indexed as soon as possible to start brining traffic. If you cleint is in a tough market, getting your product indexed and out first could make a difference.
It also matters if your content is being syndicated, getting it out there first may help SE's determine the ownership of the content.
Or if your blogging on story that other bloggers will cover too. You want yours to be indexed, ranked and attract those links first.
This is absolutely great news for SEOs that love Twitter. "What?! I'm getting us indexed!"
News sites probably wouldn't need tweets for quick indexation. I have a wordpress news blog and something surprising happened the other day. I posted with an error in the title. I immediately corrected it but later on to my surprise I saw that Google had indexed the first version. It must have been indexed within 15 seconds without any help from twitter.
On a blog and a news site for that matter, often tools are in place to rewrite the XML sitemap and then feed servers are pinged and The Google comes running over to grab this genius QDF satisfying content of yours.
After reading this, I took several subpages of a website I'm working on and "tested" this process. One the pages that I did not Twitter, the pages have not yet been indexed (only been 48 hours since creation). On the pages I did Twitter, following this process, I did find that 75% pages were indexed and cached within 24 hours. I will note that I did use the bit.ly shortener URL but I customized the URL to include the keyword I was targeting for that page. Even though Twitter uses "nofollow" on the links, I still thought this may be prudent. Not sure if it is directly related or caused any additional findings.
Also, as a side note, I tested this on a blog site that I've also had some trouble with and, to day, nothing has been cached or indexed. So, perhaps it is a practice that may work on static websites but not so much on blogs or dynamic sites? Just a thought.
I'd love to hear any experiences on whether or not the anchor text provided any value, however slight.
Thanks for the post. Do you get the same effects posting updates on Facebook? It would be really interesting to see a comparitive study.
The great thing is that Google has direct access to Twitter's database so they know, within seconds, about new tweets. I don't think Facebook has given Google access to that type of data, though I may be wrong.
It's a good test but not surprising to me. I think after some years the Twitter will be full of spams, persons who just add links and never read others... It will be interesting to do this test with the Facebook tweets too...
do you have any idea about top tweets?
I bumped into this post while searching for an answer. Coincidently, I also bumped into this around the same time - a very recent study by Eric Enge reveals tweets aren't really getting indexed by Google often.
Better late than never to this topic. I agree that Twitter does get the crawlers to your site quicker. Thank you for pointing this out, I didn't think about the benefits until now.
Twitter could be a quick start link that you need in order for you to make your wheels going and agreed that twitter helps a site to get a faster index from google.
Great Post love twitter. Its nice to see someone experimenting and providing other SEO's with the pertinent info.
Great Article.. You sure you don't have futurest in QA testing world?
Social media is fast becoming an essential part part the marketing mix for brands. Companies are increasingly using social tools to monitor conversations about their products, competitors, and industry, and engaging with their customers to build strong relationships.
Twitter is definitely getting upper hand in Google with having every single tweet being indexed by Google. It might be wise to tie your blog with twitter account so everytime you do a blog, it automatically posts on your twitter account. I have yet to find something like this for facbeook! I'm sure there is a app there, I just need to look into it bit harder.
Twitter has now moved up to the #1 position as my favorite social network!
Good stuff! While I think the correlation between indexation speed and retweets is interesting, fast indexation isn't much of a problem for most of the sites I work on. Some of the conclusions you can come to by inference are a bit more interesting to me.
For one thing, this is real, actual data supporting the idea that some sort of social media quality/authority metric is already in use by Google. Faster indexation is only the tip of the iceberg - what else are they determining based on retweet counts of particular content, particular users, etc?
Great post, just in time, had a meeting a few minutes ago discssing the webmaster tools and influence of site map submission on indexing.
Twitter is getting lots of respect from Google, las time I googled up 'cheap flights' the first result was Cheap Flight's twitter acount not their main site, was a bit surprised.
This sort of thing makes us all look good. If we can say, "You know, when accomplished in the right way our experiments show (a customer simply won't understand 'descriptive study') that 3 retweets speed up indexation. Now we're keeping a close eye on this ... blah, blah, blah."
Now when we couple that with insights from Baxter's Whiteboard Friday: https://www.seomoz.org/blog/whiteboard-friday-get-indexed-faster-with-richard-baxter ... we all seem like freakin' geniuses. At least we're offering an attitude of really thinking about this stuff and for staying on top of trends.
When we paint this large picture filled with complimentary options and present them to a client. Everyone wins.
Now with all the niceties out of the way, I have to say ... OMG WTF? You did what to figure this out? Man-alive! Granted I use a few cool tools and I toss some raw data into pivot tables for analysis and my client documentation is almost beautiful ... I simply don't have the mental bandwidth to run experiments like that.
At the moment, I admire you while while wondering what the heck makes you tick. I believe that you, some of the Critchlows and a few others will help ease me into a deeper data and research phase.
Thanks for this great info. It never hurts to gather more information about getting indexed. With a company that builds websites frequently it is always good to know was to get indexed faster and deeper. Seeing that it will take a small site (250 pages is small in the eye of Google) a day or two to get indexed, and even longer to get ranked.
Interesting assumption at the end that Twitter must jump when Google asks. The potential symbiosis between the two may be close enough that Twitter will instead push the feed data to the search partner.
As much as I abhor people using Twitter for means other than socialising, good article :p.
I'm glad you've actually not given the advice to spam people. If you'd done that I'd be at your door with a pitchfork.
I agree with some of the comments regarding the reason for doing this. It is fully dependent on what type of site you run and should it be a news site or a site that brings up to date information frequently then it`s fair to say that this technique would be useful. However, in general I don`t see why anyone would bother when Google Indexes within a matter of a minute after publishing and therefore it makes the process of using Tweeting to bring traffic a bit redundant. But for news sites it is definitely a positive step to take.
Great post! I use twitter a lot.
Great comment!
LOL! Down boy.
Twitter is getting lots of respect from Google Using Twitter can help more indexation. Many users twitter twitter for a new position or products on their own account, hoping to attract The attention and links from their followers.........................
Just a brief methods note:
Be careful with nomenclature. Unless I'm misunderstanding, this isn't an experiment, it's a descriptive study. The distinction is pretty important as the former supports causal conclusions while the latter only raises the possibility that causality might exist.
Assuming I understand correctly, your conclusion that links and tweets will "help in increasing" indexation may be misleading. The most you can say is that linked-to and tweeted pages are indexed faster on average that those that are not.
I hope these points don't seem nitpicky, but I think it's a good idea to adhere to generally accepted research standards and definitions.
Aside from that, I really enjoyed your post and will be referencing it in the future. Thanks for the great contribution!
Thanks Sean. I don't claim to be a scientific but will take your suggestion into consideration in my new post with data. Granted I didn't have too many data points but I did see a slight increase in the number of pages that were indexed when using Twitter vs only linking to the page via internal links. Definitely not enough to be statistically positive that it would do that for everyone.
Thanks for the response Casey. I wasn't really questioning the number of data points, or the statistical significance of the results. Really, my main concern was the use of the word "experiment." After taking a look at the Wikipedia entry for the term, I realize that I jumped the gun a bit.
I've always been taught that in order to be an experiment, a research study had to have certain components such as independent and dependent variables, controls, and random assignment. Wikipedia refers to these as "controlled experiments," which are different from "natural experiments," and "field experiments."
Your study absolutely fulfills the definition of a natural experiment. I was mistaken, and I apologize for calling you out as I did. Even though it doesn't fit the narrow "experiment" definition that some scientists use, your study is valuable research and deserves attention. I really enjoyed reading it, I applaud your initiative, and I look forward to seeing more from you.
:-)
I knew you'd be weighing in Sean. No statistics discussion would be complete without you.
I think you should approach the mozplex with the suggestion of making you an SEOmoz Associate and you could be the statistics editor.
Seriously.
Thanks for the compliment GNC. I actually did informally approach them a couple of times, but I got the impression they weren't looking to fill such a position at the time. If they change their minds, I'm willing to consider it ;-)
Judging by the thumbs-downs this comment received, I'm guessing that it came off as a bit nitpicky, overly-critical, trolly, or maybe just factually incorrect. I hastily wrote it early this morning before work, and didn't take the time to be as diligent with my facts, or as tactful with my words as I usually am. As it turns out, I was rather incorrect in my assertion. You can read more in my comment above.