Hey everyone, I'm Tela. I head up data planning at SEOmoz, working on our indexes, our Mozscape API, and other really fun technical and data-focused products. This is actually my first post on the blog, and I get to announce a brand new feature - fun!
One of the challenges inbound marketers face is knowing when a new link has surfaced. Today, we're thrilled to announce a new feature in Open Site Explorer that helps you discover new links within an hour of them going up on the web: the Just-Discovered Links report.
This report helps you capitalize on links while they're still fresh, see how your content is resonating through social channels, gauge overall sentiment of the links being shared, give you a head start on instant outreach campaigns, and scope out which links your competitors are getting. Just-Discovered Links is in beta, and you can find it in Open Site Explorer as a new tab on the right. Ready to learn more? Let's go!
What is the Just-Discovered Links report?
This report is driven by a new SEOmoz index that is independent from the Mozscape index, and is populated with URLs that are shared on Twitter. This means that if you would like to have a URL included in the index, just tweet it through any Twitter account.
One note: The crawlers respect robots.txt and politeness rules, which would prevent such URLs from being indexed. Also, we won't index URLs that return a 500 status code.
Who is it for?
Our toolsets and data sources are expanding to support a wider set of inbound marketing activities, but we designed Just-Discovered Links with link builders in mind.
Getting started
You can search Just-Discovered Links through the main search box on Open Site Explorer. Enter a domain, subdomain, or specific URL just as you would when using the Inbound Links report. Then select the Just-Discovered Links beta tab. The report gives PRO members up to 10,000 links with anchor text and the destination URL, as well as Domain Authority and Page Authority metrics.
One important note on Page Authority: we will generally not have a Page Authority score available for new URLs, and will show [No data] in this case. So, when you see [No data], it generally indicates a link on a new page.
You can also filter the results using many of the same filter drop-downs you are used to using in other reports in Open Site Explorer. These include followed and no-followed links, and 301s; as well as internal or external links, and links to specific pages or subdomains. Note: We recommend you start searches using the default "pages on this root domain" query, and refine your search from there.
How does it work?
When a link is tweeted, we crawl that URL within minutes. We also crawl all of the links on the page that have been tweeted. These URLs, their anchor text, and their meta data (such as nofollow, redirect, and more) are stored and indexed. It may take up to an hour for links to be retrieved, crawled, and indexed.
We were able to build this feature rapidly by reusing much of the technology stack from Fresh Web Explorer. The indexes and implementation are a little different, but the underlying technology is the same. Dan Lecocq, the lead engineer on both projects, recently wrote an excellent post explaining the crawling and indexing infrastructure we use for Fresh Web explorer.
There are a few notable differences: we don’t use a crawl scheduler because we just index tweeted URLs as they come in. That’s how we are able to include URLs quickly. Also, unlike Fresh Web Explorer, the Just-Discovered Links report is focused exclusively on anchor text and URLs, so we don’t do any de-chroming as that would mean excluding some links that could be valuable.
How is it different?
Freshness
Freshness of data continues to be a top priority when we design new products. We have traditionally released indexes on the timeframe of weeks. With this report, we have a new link index that is updated in about an hour. From weeks to an hour - wow! We'll be providing additional details in the future on what this means.
URL coverage
This index includes valuable links that may be high-quality and topically relevant to your site or specific URL but are new, and thus have a low Page Authority score. This means they may not be included in the Mozscape index until they have been established and earned their own links. With this new index, we expect to uncover high-quality links significantly faster than they would appear in Mozscape.
I want to clarify that we are not injecting URLs from the Just-Discovered Links report into our Mozscape index. We will be able to do this in the future, but we want to gather customer feedback and understand usage before connecting these two indexes. So for now, the indexes are completely separate.
How big is the index?
We have seeded the index and are adding new URLs as they are shared, but don’t yet have a full 30 days worth of data in the index. We are projecting that the index will include between 250 million and 300 million URLs when full. We keep adding data, and will be at full capacity in the next week.
How long will URLs stay in the index?
We are keeping URLs in the index for 30 days. After that, URLs will fall out of the index and not appear in the Just-Discovered Links report. However, you can tweet the URL and it will be included again.
How long does it take to index a URL?
We are able to crawl and include URLs in the live index within an hour of being shared on Twitter. You may see URLs appear in the report more quickly, but generally you can expect it to take about an hour.
Why did you choose Twitter as a data source?
About 10% of tweets include URLs, and many Twitter users share links as a primary activity. However, we would like to include other data sources that are of value. I’d love to hear from folks in the comments below on data sources they would like to see us consider for inclusion in this report.
How much data can I get?
The Just-Discovered Links report has the same usage limits as the Inbound Links report in Open Site Explorer. PRO customers can retrieve 10,000 results per day, community members can get 20 results, and guests can see the first five results.
What is “UTC” in the Date Crawled column?
We report time in UTC, or Coordinated Universal Time format. This time format will be familiar for our European customers, but might not be as familiar for customers in the states. The time zones for UTC are ahead of Eastern Standard Time, so US customers will see links where the time-stamp appears to be in the future, but this is really just a time zone issue. We can discover links quickly, but can’t predict links before they happen. Yet, anyways :)
CSV export
You can export a CSV with the results from your Just-Discovered Links report search. The CSV export will be limited to 5,000 links for now. We plan to increase this to 10,000 rows of data in the near future. We need to re-tool some of Open Site Explorer’s data storage infrastructure before we can offer a larger exports, and don’t have an exact ETA for this addition quite yet.
This is a beta release
We wanted to roll this out quickly so we can gather feedback from our customers on how they use this data, and on overall features. We have a survey where you can make suggestions for improving the feature and leave feedback. However, please keep in mind the fact that this is a beta when deciding how to use this data as part of your workflow. We may make changes based on feedback we get that result in changes to the reports.
Top four ways to use Just-Discovered Links
Quick outreach is critical for link building. The Just-Discovered Links report helps you find link opportunities within a short time of being shared, increasing the likelihood that you’ll be able to earn short-term link-building wins and build a relationship with long-term value. Here are four ways to use the recency of these links to help your SEO efforts:
- Link building: Download the CSV and sort based on anchor text to focus on keywords you are interested in. Are there any no-followed links you could get switched to followed? Sort by Domain Authority for new links to prioritize your efforts.
- Competitor research: See links to your competitor as they stream-in. Filter out internal links to understand their link building strategy. See where they are getting followed links and no-followed links. You can also identify low-quality link sources that you may want to avoid. Filter by internal links for your competitors to identify issues with their information architecture. Are lots of their shared links 301s? Are they no-following internal links on a regular basis?
- Your broken links: The CSV export shows the http status code for links. Use this to find 404 links to your site and reach-out to get the links changed to a working URL.
- Competitor broken links: Find broken links going to your competitors’ sites. Reach out and have them link to your site instead.
Ready to find some links?
We’ve been releasing new versions of our Mozscape index about every two weeks. An index that is continuously updated within an hour is new for us, too, and we’re still learning how this can make a positive impact on your workflow. Just as with the release of Fresh Web Explorer, we would love to get feedback from you on how you use this report, as well as any issues that you uncover so we can address them quickly.
The report is live and ready to use now. Head on over to Open Site Explorer’s new Just-Discovered Links tab and get started!
This is pretty awesome. I'd love to get an RSS feed of the latest links.
Thanks for the suggestion, Sean. That's a great idea!
RSS feed on tool data - now thats a great idea and it sounds relatively easy to implement too!
Great idea. We need a simpler way to use this data. Let's say I have 30 competitors I want to watch - if I have to run individual reports for each using OSE and Fresh Web Explorer, that could be a couple hours of work per day/week. With a central campaign feature, or RSS feeds, you could do it in minutes.
that is indeed a great idea
I'm using https://page2rss.com/ combined with IFTTT. Seems to work pretty good so far.
Just finished creating some bookmarklets for the new tool - check 'em out:
We've got 3 bookmarklets for "FDL for This URL", "FDL for This Subdomain", and "FDL for This Domain".
Settings will default to all links from external domains, but the bookmarklet is easy to tweak if you prefer another setting.
We've tested these in Chrome, Firefox, IE, and Safari, but please let us know if you encounter any quirkiness. Email is tools at contentharmony dot com.
Thanks for assembling those bookmarklets, Kane! They'll definitely make it even easier for folks to use the feature!
nice Kane.
This is just straight FIRE, great idea and VERY nice addition to a great too.
Absolutely love this - so much potential! Any plans to integrate fresh links into the Moz API?
Matt
Hi Matt -We are looking to make this kind of data available as an API in the future. However, we have a few options for how we go about it. That's a big part of why we released this beta, to get an understanding of how this data is used by our customers to help inform the decision.
Why does the video style remind me of Apple?Like the optics - great news. :-)
That's courtesy of Elijah, our new in-house videographer. You'll be seeing more of that. Glad you like!
Glad you enjoyed the video! Keep an eye on the blog for more to come :)
Wooah! Just awesome new! Amazing feature) Thanks guys!
This is really solid. A nice way to stay up on fresh link opportunities and help me to continue to come up with interesting content ideas. Nice.
Big fan of tools that give this kind of data - so thanks!
One suggestion though, is adding a way to filter links that are affiliates. For companies that use large sized affiliates like Polyvore it's hard to see anything of value without manually checking thousands of links, even with this tool.
Hi matt.ridout -Thanks for bringing this up. I have worked with a lot of affiliate systems and had considered this use case. Unfortunately, it wasn't something we could address in this release. However, for future releases is certainly functionality we could consider. Do all of the affiliate systems you use include an appended tracking parameter such as ?affid=34753 ? I'd love to get more details on the type of filtering that would be useful to you.
Whoo hoo! Outstanding news! Thanks so much for providing such awesome tools :)
Thanks very much, this is a great feature, especially for keeping up with what my competitors are up to!
Fantastic news! I love data - and @Gyi - I don't think this will lead to "more twitter spam" - basically anyone who will find a Moz tool to check links is already using Twitter to create links in the first place.
Going to test this out - looks excellent!
While we're on the topic of tools - can you guys add FWE to the Chrome bar when you update it?
I thought that too. Then I saw folks tweeting URLs today in an obvious attempt to get them in the "fresh index."
That's a great suggestion on the Toolbar. I'll get together on that with Jon, the product manager who owns the toolbar.
Hi Matt - thanks for the suggestion. Over time we will see all of our indexed data in the MozBar - including Fresh Web and Just Discovered. They are really useful at the page level and we know you guys find the toolbar really useful as a shortcut!
The MozBar hasn't had too many updates in the last year or so but I am working on that :) Would love to hear any more feedback if you have it - watch this space for updates!!
Is it more than just links/pages shared within an hour? I found a few "just discovered" links that were a year old on popular domains, but on internal pages that were unlikely just shared (though I suppose it's possible).
Or, is the just shared link the basis for a quick domain crawl and index?
Old URLs are often going to appear here, but only if/when they've been shared. The only data source powering this to my knowledge is the stream of tweeted URLs, so it means that somebody shared that URL on Twitter in the past few hours.
Tela - please correct me if I'm wrong (I know you're way closer to the engineering side of this project than I've been).
Hi Bill -
Rand is correct. We index all tweeted URLs (with the exception of those where we have been blocked by robots.txt or politeness, which we respect), and keep them in the index for 30 days. After 30 days, URLs expire out of the index. If a URL is tweeted again, we crawl and index it again, and the 30 day clock starts over.
To answer the second question, we don't do a domain crawl. We crawl the tweeted URL and all internal pages that are linked from the tweeted URL. The result is a graph of URLs that are accessible from freshly-shared content, many of which are not in our Mozscape index.
Great addition to OSE - Ties in nicely with https://freshwebexplorer.seomoz.org/ for identifying new opportunities.
Looking forward to delving into this more but already found some new and exciting prospects.
I'm happy to report that I broke through my 200 points today and my nofollow profile link dropped in within the hour!
This is amazing really amazing, I have seen some advertizements saying they provide alert service for each new back links and new reputations.
Thanks dear Tela Andrews for your post
Pinterest. Would love to see newly discovered links from there. Thank you!
more instant success here for me! I checked a competitor domain earlier today, found a possible link source, got the link, and it showed up in the Just-discovered index within the hour!
This is a pretty slick feature. Thanks!
Great! More Awesomeness from the Moz team! I love your tools! This and Fresh Web Explorer are two great new additions to your tool set.
Interesting data, well done! The ability to set up searches to be sent as weekly emails (kind of like a google alert) would be awesome. Don't know how possible it is but some way to do geographic restrictions might also be nice, if say you are looking for commonly used english phrases or brandnames but more interested in a non english speaking market.
Great suggestions all-around, lynnch. Adding these to the feature request list!Cheers
Any plans to add links from other social networks into this in the future? G+, Facebook and Pinterest would the ones i'm most interested in.
Hi Dave -I'll add your requests to the list of potential data sources. It sounds like links from Pinterest in particular would be really interesting to a lot of folks. Thanks for your suggestions!
Thanks to Tela and team, this is a very useful tool, Pintrest would be a great addition!
Thank you Tela. I like Dianemahan suggestion for pinterest.
Thats awesome feature, i would definitely try for it...Thanks
This is so cool! Thanks Tela :)
Works well - cool idea - a little special benefit :)
Thanks for sharing the new open site explorer with us. The open site explorer can help to discover new links with an hour of them going up on. The just discovered links report is driven by a new SEOmoz index that is independent of the Mozscape index, and is populated with URLs that are shared on Twitter. This is very helpful to discover the links easily.
Moz, thanks for this feature! I have found it very valuable to discover the fresh content referencing my site an authority as well as seeing what types of sharing are occurring.
Hi, Tela.
I think this is cool.
But i need to get just-discovered count by using API.
Is it possible?
Hi,
They say you will eventually visit the site out of necessity and the time has actually come for me to do the same. Yes, as I have new blogger and presume that some of the blogs are yet to show links which necessitated me to search for this feature. My blog is
www.socialchaal.com
I am going to look for more information to complement the knowledge that you have given us, it is very interesting thank you very much
Any progress with the API?
Hi Digital360 -
We are not currently planning to release a stand-alone API for Just-Discovered Links at this time, but are planning to include this data in the API for our next-generation web index. It will be a phased release, and the plans are still being solidified, but it's safe to say that this would not be available before 2014. Thanks for your interest!
Thank you. This is a top addition to the flow of information I need. Reassures me that our recent video launch is really effective, and other work is too!
Hi,
I have noticed in this new tab that several links (not from Twitter) are showing.
Will the links that have 'just been discovered' move to the Inbound links tab after a while?
Thanks
Hey Hamit,
Currently the Just Discovered Links and Mozscape indexes are completely separate so 'just discovered' links will need to be organically discovered in order to be included in the Inbound Links tab. Our Big Data team is working on a project right now to improve the way we scheduled fresh links, however, so look for this to change in the upcoming months as we work on scheduling much fresher link data in our Mozscape index!
Hope that helps!
Carin
Hi Carin,
Thanks, that helps a lot :)
Hamit
Hi!! Excellent tool, totally love it!
Does it work for social signals and their links only! or does it work also on general web links?
Regards!
Hi JesusD - this is not a tool based on a general web index. Links are coming specifically from those shared on Twitter. I hope this answers your question!
I don't see new links for my domain. Why is that?
Hi Nagar -
I did a search for the domain listed in your profile and found two results. The query is below. If you're not seeing many links it could be an indicator that the URLs that link to you are not shared on social media. One takeaway is that there could be an opportunity for you to get more links from sites that get more social shares.
https://www.opensiteexplorer.org/just-discovered?site=www.phraseexpander.com
I hope that helps!
This has been very useful to play around with. Could you guys add other discovery channels as well, such as Facebook, Google+, Pinterest, etc?
Hi Takeshi!
I'm glad you've found this report helpful. We may add other sources of fresh links in the future, but these are not our priority right now. In the near term, we're focused on adding these links to our Mozscape index.
This is pretty cool but lot of your customers like us will probably need a way to get an automatic update to their inbox - RSS/SMS/Daily Email report etc
Awesome!
Finding the just achieved links is very useful now. It really works well. I doubt seomoz tool finds the links before google bots.
Hi,
nice functionality.
You say it shows links back to 30 days, but I noticed that current data go back until 1st march.
is it ok?
thanks
Hi T20,We back-filled the index as far as possible during development, which wasn't the full 30 days. Now that the beta release is out, the index will fill up to the full 30 days. Thanks for calling that out specifically, I neglected to mention that detail in the blog post.Cheers!
Started playing with it already - it's pretty cool!
"We also crawl all of the links on the page that have been tweeted"
Do you follow links and crawl linked pages, or only tweeted page gets crawled?
Also, it appears that report shows only the first link to my root domain from one page. On one page, I have two links pointing to different pages of my site, but when I search for links to pages on my root domain, only first link is shown.
Congratulations to the whole team for their hard work!
Hi Bojan -If you download the CSV report, it has multiple links from the same URL. We are also looking to surface this across all of the reports on Open Site Explorer in the future.
Thanks for the answer
In other news, Twitter spam spikes. In other news, that's not news...
Nice new feature. Looking forward to checking it out further.
First off, indexing every hour is pretty incredible, thanks for that. I am excited to dig into this! Thanks!
Good , I will try this and see how it works with some sites that i wanna see progress.
Thanks mate :)
Amit Aadarsh
Wow! I'm in love with this new option. Thank you Moz Team.
Really it’s a good feature for OSE. Thanks to SEOmoz Team.
Well, this stuff would be the Tool that everyone would love to use. I would be glad to follow the next updates for this one. I would like to learn more about this.
Thanks!
Hi,love this feature, also because it seems - from few test I did - that is a great way to discover potential critical bad links from sophisticated scraper sites.
Brilliant, thank you!
Thanks for the heads-up on this new tool available in OSE. Definitely going to give it a spin!
Great addition to the tool!
What would be an even more awesome addition would be if it was possible to filter the results if the link has never been seen before. I'm seeing old links showing up because someone tweeted an old post - so it means having to check through them all.
Hi Code Computerlove -I completely understand that would be useful. That functionality wasn't possible with this beta version of the feature, but I'll definitely look for ways to provide this in future products or versions of this report. Thanks for the suggestion!
Love it!
Ace. This would be great to get instant feedback on how viral a piece of content has gone. If this data could be injected into the Moscape Index in future too that would be even better.
Great data. Love it. Thanks SEOMoz. BTW the Post Analytics is very interesting as well. Cheers Moz team.
This looks like an awesome way to keep track of how your content is getting shared. I'm loving the new SEOMoz tools from the last few weeks.
This is great;-) Awesome new tool. Thanks
good fresh link :)
Nice move SEOmoz, something like this was needed as Majestic SEO seems to be killing the fresh link data at the moment =)