People talk a lot about APIs in the SEO industry (me especially) - the tools you can build with them, the competitive analysis data you can access, the reports you can automate. However, we tend not to discuss the wider picture, the thousands of APIs out there for other things, and, most importantly, the profound effect that APIs are going to have on the web, and thus the SEO industry, in the coming decade.
I believe that over the next 10 years there will be a huge decline in the number of users visiting websites, and that APIs (and structured data) are going to play a pivotal role in that shift.
To understand why I think this, lets take a look at the evolution of the web over the last 20 years.
(Note - I'm not going to explain APIs in this post - but you only need a broad understanding of them to get my point.)
History of How the Web Looks
The (public) web has been around for a little over 20 years, and in that time it has changed a lot. The first website, built by Tim Berners-Lee, looked like this:
(see it here)
With black and white text and no images, it was a pretty static experience, but was the birth of something amazing. Over the next 5 years, although the web grew and moved forward, it didn't see any drastic changes. Over the next 10 years the web got more colourful, with images appearing and layout being improved:
(Image courtesy of archive.org)
However, it was still a pretty static experience, and a long way from the web we know today, which looks more like:
Good old Bear Grylls. His site (which he updated this week!) shows off animation, high-res imagery and video, is interactive, has embedded social media, and is representative of the dynamic web we know and love today.
Obviously, this was a crude history of the web, but my point here is that in 20 years the web has accelerated from something akin a simple set of interlinked text pages to an interactive multimedia system. Let's tie this together with how the web is consumed and searched.
History of How the Web is Consumed
We've seen the web has become a far richer experience to interact with in the last 20 years. We can see that it has also undergone drastic changes in the way that we consume it. In the beginning, it was almost universally accessed via desktop machines:
Some readers won't even remember the days of having to turn on the external modem, dial up to connect and wait whilst the modem sung its song as it connected for a while. Using the web back then was something you made a slot of time for - you'd decide you need to check your email, and you'd head off and maybe spend 30 minutes or an hour using the painfully slow connection, before disconnecting and turning off the modem.
That slowly changed, and by the early 2000s you were quite possibly looking at the internet on a laptop:
(Image: Wikimedia Commons)
Your laptop maybe had a built in modem giving you 'mobile internet' you could use by a phone line, but we were starting to see 'broadband' connections and the advent of wifi. This was the beginning of always being connected for some people, but still the internet wasn't truly mobile, and there certainly wasn't widespread availability.
Nowadays, the majority of readers here will be accessing the web on their mobile:
There is prolific availability of wifi, most of us have 3G internet built into our phones, and using the internet is completely without thought; on the occasion that I'm not connected I'll inevitably try to do something on my phone before remembering "Yep, that also requires internet."
We have moved from interacting with the web via infrequent and prolonged interactions of minutes or hours to many short interactions of only a few seconds. In 2002, the average internet user used the web for 46 minutes a day, but in 2012 that has gone up to 4 hours a day. We're using the web for more time, more frequently, and in shorter bursts. Using the web has moved from being a task in and of itself and has become ubiquitous tool that we don't even notice.
History of How the Web is Searched
Finally, let me briefly discuss the way the web has been searched over the last 10-15 years. Very broadly speaking the general process, which has been changing in the last 2-3 years, has been searching in a generic fashion to searching in a specific fashion. Crudely this could be thought of as 'searching to find the place where you can search for the answer'.
You traditionally have started with the '10 blue links' result, which has taken you to a place where you could often refine your search in a context specific fashion. What do I mean by context specific? I mean that this specific search allows you to specify attributes unique to the type of thing you are searching for.
For example, in the image above, the search page on Amazon allows me to specify attributes such as author, ISBN and others which are specific to the context: book. Whereas the search box on Google / Bing is a single text entry box to cover all types of search for all types of results.
A Shift in How the web is Searched
However, we have been seeing a shift in this generic -> specific model. Google has started to detect the context of your search from keywords, and then provide additional context specific inputs to refine your search. The classic example is the hotel search:
Now, beyond the general text field, Google allows me to enter a Check-In and Check-Out (attributes, unique to the context: hotel) date to refine my search, and see real-time prices from a variety of vendors. What is notable is that these prices I'm seeing from sites such as ebookers.com are right here in the search results - I'm not visiting their site to see them. I'm not seeing any of their marketing bumpf, any of their special offers or deals, I'm just seeing their price.
Google has also began customising the display of the results based on context, with various examples from maps to weather:
Rather than the 10 blue links of days gone by, we are increasingly seeing the a context specific SERP with a customised display. With emerging search technologies such as Siri, this is often the norm rather than the exception:
With Siri, there never was 10 blue links, but only ever context specific results, each presented in a fashion appropriate to that context.
Siri's weather results above are provided by Yahoo!, but again the user never visits the Yahoo! Weather site. Likewise, the results about the distance to the sun are from Wolfram Alpha, but the user never visits their website. The results are being server via APIs, allowing Siri to leverage Yahoo! and Wolfram Alpha's services directly.
Decline of the Web
If we look over this history of the web we can see that the web has become progressively more dynamic, progressively more interactive, and richer and richer. Along the way internet use has become increasingly mobile, increasingly ubiquitous, used increasingly often and in shorter and shorter interactions.
You'll notice above that I distiguish between 'the web' and 'internet use' - not all internet use is via web pages. The internet existed for many years before the web was invented, where email and bulletin boards reigned. We can see above that more and more often APIs are serving people's search requests in place of the web.
Beyond search, wildfire adoption of Apps in the last few years has seen another avenue which has replaced people web use with something else; people are Facebooking and Tweeting, doing internet banking, finding routes, sharing photos and more all directly via apps instead of via the web. Masses of these apps are using publicly available APIs, with others using private APIs.
If we extrapolate from the changes to the way we consume the web, the shift in the way search results are being delivered, the uptake of apps I can't see how there is any conclusion other than the web will continue to give way to these other technologies.
A Final Thought
Will the web die? I don't imagine so, and certainly not any time soon. However, it is going to be ever more superceded, and progressively relegated to be where your marketing lives, and less where your customers interact with you.
The majority of the services that will replace the web will be connecting via APIs, whether they are public or private. Having a private API, with an App and other ways to access it is going to be increasingly important. Having a public API allows other people to build on top of your platform; Paul Graham of the startup accelerator YCombinator says "APIs are self serve business development" (Paraphrased from this tweet).
I can imagine a lot of you are thinking 'We don't need an API!' / 'What would we do with an API?' / 'We have a website, so no need for an API.' These all sound strangely familiar to the same arguments we heard 15 years ago with the web. In the 90s hoardes of companies were unconvinced by the web, but later came to regret it. It's how companies such as Nissan ended up spending millions in later trying to secure their domain names (nissan.com remains owned by a small computer company with a handful of employees), and how other companies went out of business to their competitors who did embrace the web.
I'm not sure that all companies need an API, and we are in early days still. We're going to see the landscape change more still, and certainly many smaller companies might not need an API. However, my point is that companies need to be thinking about this, we as SEOs need to be thinking about how this will affect the industry over the next few years, how we can be forward looking to help out clients, and I'm mostly hoping this post kicks off some discussion about it all.
Interesting post, I agree with some of the concepts but its too hard to say what will be popular in 5 years or even 10 years time. 5 years ago Facebook was not really that popular, Myspace was more so the dominant site. The thing is some people will take a while to adapt to technology, older demographics will start using Facebook and Search (these demos have huge uplift atm) and not want to learn new methods, but they will adapt over time. The other day I was also talking to some one in our industry who was still using dial up internet, so things do not change =)
That been said I agree companies need to start thinking about API's but that been said some companies are just starting to think about social media...
Hey James,
I absolutely agree - predicting specifics about the future is always notoriously difficult. The bigs things seem to be the things that nobody considered (which makes sense).
However, you can often predict shifts and broad changes in direction; in the 90s some people thought the web was a fad, but others believed it was going to be critical and embraced it.
I am not sure that the specific examples in the post will be true, but I do think that the general shift towards other means of interacting online rather than solely via the web is going to continue.
If we extrapolate far into the future, we can imagine I'm at home and say (aloud to the computer present throughout my house) "I'd like to get a new carpet. What are my options?", "Well, Tom. Scott preferred CheapCarpets, but both Felix and Sophie recommended Carpets2U. Carpets2U have an offer on at the moment!", and I buy right there and then.
The web will die at some point in the future, that is a certainty. How long that will take, I don't know. But I do strongly believe that most future technologies are going to work via APIs, and having an API even prepares your service to be utilised a new technology you hadn't predicted (for example, Siri).
So - whilst I can't predict the exact future, I think a lot of the predictions would converge on APIs being a common thread.
Thoughts?
I wanted to add, as a programmer, API's aren't all that new. They've been around since the dawn of object-oriented programming (OOP). I'm not even old enough to know what it was like before OOP (... <sing> yeah, you know me. </sing>)
Tom has it right. As the web becomes more integrated - Siri, reviews, rich snippets, single logins - APIs will be more important. Just as UX is important for people, APIs are important for programmers.
Think of API's as UX for programmers.
Hey Tom, I like this concept a lot. I've recently started using the Mozscape API and have found it easy to use and pretty painless. Along the way I've found some documentation needs to be updated and not all results are consistent (page titles are sometimes incorrect and/or missing). It takes a lot of resources to build an API and even more to maintain it.
In some cases, the benefits of having an API more than outweigh the increased maintenance costs. Twitter benefited greatly by allowing third-party developers access to its API. But not every company will be in the position to have the resources to build the API and also maintain it. APIs can be very beneficial to the companies who use them well so I do believe it can provide competitive advantage. For those who don't use them well, I think it provides something else for customers to complain about.
While I like this concept a lot, I don't think it will become mainstream - mainly because maintenance of an API doesn't naturally exhibit ROI; the ROI is based on the success of third-party developers. What do others think?
Hey George,
You raise an interesting point, and one I didn't touch on really in the post a lot, but have thought about.
In their current form, you are absolutely right - APIs don't represent a positive ROI and aren't accessible to a huge number of smaller companies.
However - think about microformats for a moment and a common vocabulary such as schema.org. Not an API at all, but it is structured data that provides some of the lighter benefits of an API, and shows the engines co-ordinating on a standard.
I am hopeful that in the future we'll begin to see some API standards of some sort emerge, that will start to provide some off-the-shelf type features, making them accessible and a positive ROI move for many.
I could also imagine that many of the CMS systems start providing a configuration screen for APIs, allowing companies to click to configure such a service.
Thanks for raising this point! Let me know what you think. :)
Hey Tom,
Interesting thought about CMS's using API configurations for site content. I don't know that an API is appropriate for certains types of sites (blogs, news, and so on). This content is probably better shared with schema as you described and happens already today for some sites.
Also, CMS's will likely need to create content that is self-documenting in order to provide useful APIs. Which would be nice, because such content would be more accessible which is great for users and crawlers. Overall it sounds like a great idea and is limited only by the costs involved with creating and maintaining such a system.
Maybe this idea is the recipe for a better CMS.
I think you can definitely calculate ROI of an API.
Signups with and without an API connection:
When you sign up for Spotify, you are given the option to sign in using Facebook or create a regular login. If you A/B tested that landing page, would you have gained more or less users if they had to create a login verses using their Facebook login? I'm not Spotify, but that Facebook login is really convenient. I bet they get more signups because they integrated with Facebook.
On the other side of that world, how many people would sign up for the original website (ie Facebook) because they wanted free music?
Let someone else do the marketing for you!
Think about it.... that "sign in with Facebook" button is branding! That button is everywhere reminding me that I should be on Facebook talking to high school friends I want to avoid. Other people are doing the marketing for them! ZERO effort!
What if Facebook didn't have "sign in with Facebook". Nobody would add that distinctive blue button added to their website. No branding. Someone else's brand would be there.
An API may be necessary but is definitely not sufficient.
Your website needs to gain users in the first place to gain popularity. That popularity pulls in other developers who will use the API. Those developers will add your product to theirs which only adds exposure and popularity to your website.
Putting the technical infrastructure there to start will put you in a much better postion as you build your business. While your competitors are thinking, "people are asking for an API," you already got one.
But unless your business model is the API you need to get your underlaying business model working first.
(Edited: Attempt to fix how paragraphs were formatted.)
Definitely. Social integration for sign-in has been growing ever since OpenID. Check out what JanRain has been doing with the technology too.
Also, Mixcloud (Mat Clayton's work) is an excellent example of a business model hacking growth through intelligent API integration and built-in viralization.
Chris - thanks for jumping in on the comments! :)
This is a really great point:
"An API may be necessary but is definitely not sufficient."
Absolutely - people certainly are going to need a strong web presence for any online business for the foreseeable future. I don't expect that will change anytime soon. For example, even a successful app-based social network or other app needs a website even if it is only for marketing / info / support.
I think that might change, but in a way that none of us can currently imagine / foresee.
Great point. :)
I totally see what you're saying Chris.
In your example, Spotify is using Facebook's API not their own. The ROI of Facebook's API is measured by how successfully Spotify (and other sites) are able to implement it. This is that third-party metric I speak of that is hard to measure. It's being used because Facebook made it easy-to-use.
I do see the value, I just think many businesses won't. Great points though :).
Edit: paragraphs formatting fun
Very insightful post Tom and really interesting to read about the decline of the web.
It's a pitty you couldn't make it to Brighton SEO but Will did a great job presenting your deck.
This is perhaps a Top 10 post of all time on this blog. I'm not kidding. At least it was for me.
Because this article confirmed the reality about the present (and future) of search.
Here's the reality of the situation for us... it's no longer good enough to show up on the first page of search engine results to get organic traffic... because the new normal is there's only one search result!
For example, ask Siri, "Who won the Yankee's game today" and she returns one search result. Just one.
Ask Google Jelly Bean, "How many stones were used to build the great pyramid in Egypt?" and it returns just one answer.
Search good 'ol Google.com for just about anything and you get a mashup of blended results - most pointing to a Google-owned property. Or, Google flat out steals our content and displays it as their own (with a gray, non-underlined barely noticeable link to the source). At the very least it's been documented on this blog that Google is showing just 7 results per page (instead of the old 10-listing standard).
But there might be good news here...
... And Tom hit the nail on the head - you need to open up your content. And enabling your data (probably stored on your site in a mySQL database) via API is your last-ditch way to get your link to show up on that one result.
Yes, an RSS feed is helpful. But it's very limited (depending on how you set it up). And Google is shutting down its Feedburner service in October - that tells me RSS is dead, too.
API is WAY more open... it literally allows these search engines to access your entire library of data in a single, machine-ready "call."
For example, if you blog using WordPress, you can get free open-source software that "hooks" into your blog's database. I show you where to get it here (I even include an example for you to see).
For me, I saw the handwriting on the wall a LONG time ago.
SEM is different today. It changed in plain site (Fabian style - gradually over time... like the frog-in-boiling-water story). And I no longer rely on search engines to send me traffic anymore.
Dear Sir,
I am just starting an online IM business. How can I create an API and make it mobile. From my research this is the future. thank you
saragee
It is really amazing that you have comparing website outlook of todays and past,there is wast difference between old website and todays website because nowadays we are using animations and flash effects for making our website more attractive. It is really enjoyable to know how web search look like in past and how it will search in future. Amazing research tom, i think you have posted brilliant concept. I don't think so web will never die. Before completing my comment i want to as about some external query regarding API is that it is good for SEO to have a google maps api as the primary source of navigation?
Looking in retrospect via all the chronological data you presented, we can see that the web is constantly evolving. And as more time passes by, we can talk about revolution, rather than evolution. The way users interact with the web, with the material shared- it opens all kinds of possibilities.
And while some are firmly convinced that the gold era of the Internet has ended long time ago, we are in fact on the brinks of another one, which in my contention is going to be even more open for opportunities, though for those who are willing to understand the radical shift we are gradually experiencing.
I love trying to understand the big picture even while it is still being painted, thus this post was a great read.
Slavko
I think the 'era of the internet' might be over in the same way as the 'era of the television' could be considered over. There are more TVs now than ever before, but it isn't the 'era of television' any longer mostly because we take them so much for granted.
I think the same is happening with the internet. We take it absolutely for granted nowadays, and it is completely embedded into our lives, so we stop thinking about it.
Wow, a really good read! It's interesting to think how the web has changed over the past years!
Interesting read, I bore my girlfriend silly about why companies didn't embrace the web at the start and how they are now struggling. Will they learn this time around?
I don't know much about APIs can anyone suggest a starting place to read up on them?
Thanks
Stuart
I mean for a very top level view on what is API's you can start with Wiki Pedia - https://en.wikipedia.org/wiki/Application_programming_interface
And Youtube - https://www.youtube.com/watch?v=4JjN54aaF74
But for further reading just search around online some good reading on Quora and other online sites, their is a lot of information, I know it can take a bit to get a grasp of the concepts.
I haven't personally tried it but based on the other courses https://www.codecademy.com/tracks/apis could be a reasonable entry point.
It is really interesting to think of how the web and how we use it have evolved in such a relatively short time frame. I like to joke that we live in the future--we have all the collective information of the world in a PHONE! It never ceases to amaze me how it has grown and changed and will continue to do so. I imagine that the web 20 years from now will be even more amazing.
Thoughtful post. I like big picture thinking about the future. I imagine this effort will generate a large response.
I find myself agreeing with your premises. To my mind, internet evolution is now being driven by hardware capacity for personalization and mobility and the API is a key to that growth.
Is the web dying? I don't know. The functionality of the web I was introduced to in 1994 to the internet I know today could indicate the web is "dead" now, replaced by a more powerful, functional and portable entity which was never given a new name (other perhaps than the rather unromantic moniker of "Content Management System").
Interesting point. However, we're still all interactive via 'web browsers', so I think we could say it has evolved significantly. Just I believe that in various areas it is being overtaken by other technologies.
The web isn't dead. Yet.
APIs aren't designed to replace the web, APIs are designed to help business be agile as Intarwebs change!
You want to be agile as the Internet shifts, right? Think: How fast is technology moving now? How about "warp speed?"
Part of the promise that APIs it's the engine that powers the front-end. Companies like Quora built their API's first. Why? They start with the API, then build the website in front of the API.
Oh! Mobile is big.. instead of building a whole new website just create a new front-end for mobile using the same API. Boom! Same content, new face. Done! Oh, the competition is still building a completely separate mobile version? Sucks to be them.
Big ups to Tom. He's getting people to think outside the box (I hate that phrase but it fits perfectly here). What will the web look like in 5 years? That's the kind of forward thinking Tom has.
Nice & thoughtful Post Tom,
I think API is the most important thing for website to be publish it.
Hi Tom,
Of course I agree with everyone about what a thought provoking post this is - what got me thinking was what this might mean for advertisers and brand awareness.
If APIs evolve in this way, then like TV, it will allow users to access all the content they want whilst avoiding advertising (brilliant, can't wait - isn't the user experience of the web a million times better now there aren't pop up ads everywhere). As you point out, open access to data and innovative user focused applications of that data will become even more of a branding decision by companies, in the same way that early adoption of the web was a signifier of brand.
But what will it mean for the reach and ROI of Google Ads, if users can access content through APIs and avoid advert filled SERPs? How will that effect the million of small advertisers that (over) rely on Google ads? Perhaps the most interesting question is how will Google evolve as users want different things from search?
Definitely not dead then . . .
Interesting post, Tom. I have to say that the more I read this post, the more I got convinced to the easy possibility of this happening. It is true that people are influenced by the latest trends (smartphones and tablets) and things being mobile and intermittent use of internet access. API's are steadily climbing and more and more developers are being creative with their products to serve its users. Cheers!
Hi Tom,
Brilliant thoughts!
Just watched the "New" MySpace video (https://new.myspace.com/play) and it looks like their strategy involves tapping into loads of structured data. Could open up some fun SEO strategies moving forward.
Great and informative post Tom! I really liked the way you explained how the web was in past and how it can look alike in future. From a black in white website to a color website, from a static simple website to dynamic graphic and flash website, and obviously from a desktop compatible to our mobile compatible.
I don't think so web will die in future in fact sure to have new unique look of web. :)
Very informative post!
I hope now they dont change again the rules for the next 5 years just now that i'm used with this!
Seeing how web has changed to its current form now, it makes me more excited as I want to know how far the web tecnology will go. As for API which is very common among programmers, the use of API will not replace web instead it will make the web technology take one step further.
Very informative post! It was great to know how was the previous web and what will be its shape after 5-10 years from now. Definitely APIs are going to be crucial and every website owners need to be thinking about it.
Many post on the internet on this topic, however in this article I have been able to better understand the subject.
Shame that you didn't make it Tom, although you left it in safe hands (you know Will was calling you names, right?).
Definitely one of the presentations to get my brain racing, not what I was expecting from the title at all. It's amazing how quickly people have forgotten what a small part of the internet the web originally was... and will likely become again. It gave me a real moment of wondering "Why haven't I thought about this before?"
I'm not quite sure what to do with this information at the moment. I'm certainly eyeing our projects thinking about which might have potential as an API now though.
Hope you're better.
Great insight looking forward. Regarding mobile, many apps are communicating with API's to display info already. So it would be interesting to see if search actually declines in the future as more pragmatic apps develop that are customize for the user. By then the smart phones are programmed to your behavior w/out going to a search engine to look things up.
Again great stuff Tom!
Tom,
Great article, I really enjoyed it. I agree with previous comments on how APIs have changed how we interact over the years, especially as content becomes more dynamic. It's also interesting to see the difference in websites. I think that many CMSs are starting to have more and more API integration either "out of the box" or through plugins, which is really interesting. For example, I noticed that there was a MozScape API plugin for WordPress, unfortunately it is completely dated and no longer works.
Its now clear to me the role of APIs but its also important to anticipate things especially but now that we are going through a fast pace of modernization. We have to consider this idea so that we can adopt and still make it on top. Change is inevitable indeed.
Sorry I like this subject but the content seems to be a little off the topic. There was not much information about the specific benefits of having an API for the current being and for the future. Even though I enjoyed reading the history of the internet & search industry, I'd love more to know the real deal about API. Thanks.
Really interesting and informative post. It´s difficult to say what would the future API but i am agree with you and i also think about the future changes in the API. Everything would be fast and easy to use because it will be the demand for the such period. And at last you have given beautiful insight on API. As you described the history and changes comes as per the time really useful for everyone.
Thanks again and keep it up............
A very thought provoking article, you can see the thing moving in this direction already, especially in SERPs and in mobile device UI. It will be interesting to see this evolve and the effect it will have on the web as we know it.
Yuoo, great insight about API, Now a days API makes life better, going towards much advance....
Cheers!
Manoj
Very interesting and very enjoyable Story....Thank you so much for this kind of interesting post...!!!
The Google Feedburner APIs have been officially deprecated as of May 26, 2011 will be shut down on October 20, 2012.
There's also a list of others which Google is shutting down.