In the near future, I think Google Now could tell you are pregnant or diagnose you with a medical condition before your doctor ever could. Humans are great at recognising patterns but only if we know we are creating them or where to look. Remember the Target story of how they knew a young girl was pregnant before she or her father did? Increases in technology like smart watches and the trend of "the quantified self" mean messages like being told you are pregnant aren't impossible in the near future. So how do we go from weather reports and traffic updates to a medical diagnosis?
Strings-to-things, things-to-actions
When Google, Yahoo and Bing announced Schema.org in 2011, search engines were still in the strings-to-things phase. In my opinion, Google, in particular, are already moving on from that goal. The most recent addition to the Schema.org vocabulary is actions. See, the Schema site for more details or my SMX Munich deck for more details.
In my presentation, I made the point that the future of structured data isn't about understanding what a thing is, it's about understanding what a thing can do. If search engines can understand what your website, app or other interfaces can do, and they can understand user intent, they can match queries to the best place to do that action. How does Google know what we want to do?
Actions-to-anticipation
Many people have said that Google wants to become the ultimate personal assistant. Things like Google Now and conversational search reinforce this standpoint. However, a prerequisite for that position is the concept of time. For a computerised personal assistant to be truly as useful as the real thing, they need to be aware of the past, the present and more importantly the future.
Historically, Google and other search engines have dealt with things from the past. Webpages by their nature are in the past, or at best, live. This makes the anticipation and initiative that you would expect from great personal assistant difficult for Google. They have very little data to predict what you might want to do or are going to do in the future. Gmail and Google calendar are the two most obvious ones that come to mind (if you use them).
Forgetting privacy or intellectual property for a second, imagine Google had access to every app on your phone and the data within it. What might they be able to know about you?
Just the apps above could give Google access to:
- What music I've listened to in the past
- What movies I've watched
- What I've been eating and drinking recently
- How much exercise I do
- What articles I might read in the near future
- Flights I have booked
- Houses I might want to buy
Google Now - An IFTTT for your life
While I was in Munich, I saw an announcement that Google had opened the Google Now API to a selection of hand-picked, third-party apps.
This got me thinking. I do not know what the relationship will be or what data Google would have access to but one of the apps that have been accepted to work with Google Now is Lyft. The example Google gave in the article was a generic prompt to order a cab. For example, you arrive at an airport and Google Now might push you a notification to get a cab:
Some more examples
While the Lyft example above is interesting, it made me realise that allowing apps to talk to each other via Google Now would essentially turn your smartphone into an IFTTT for your life. So rather than a generic Lyft alert, what if they combined a few apps? They could use my British Airways app to see I have an upcoming flight, Google maps to know when I've arrived in Munich, and my Gmail account to see where I am staying. There are probably specific hotel apps they could use too. Using this, rather than getting a generic get a car card, I get one that's already personalised the quote to where I'm going.
Anticipation to diagnosis
The ultimate personal assistant would not only tell us what we expect, they would tell us things we never thought to consider. This would only be made possible by advanced pattern recognition, anticipation, and initiative beyond the possibility of a human.
What patterns do you already create but don't currently correlate? If you feel sluggish or tired on a Thursday, we do not necessarily correlate that to something that you may be allergic to that you ate on Monday. Many people spend years with conditions such as gluten or lactose intolerance but never make the connection between what they eat and how they feel. Humans cannot easily track and analyse lots of data like that, computers can.
So how can Google tell you are pregnant? I am not a doctor but I suspect like the Target example, there may be early signs of pregnancy that we do not think about at the moment (biological or otherwise). For a start, there could be a process of first increasing the priority that a particular pattern receives. For example, there may be lots of small things that people change before trying to get pregnant. If you're using a lot of different apps combined with hardware like heart rate monitors and blood pressure monitors, it wouldn't be too difficult for Google to take an educated guess. Just using the information in the Target article we know people do things like:
- Change their diet - This would be easy to see through apps like MyFitnessPal
- Change their buying habits - Amazon app or other store apps
- They may do more exercise - Several places they could get this
After all of the above, let's not forget Google knows everything you've searched for online and your browsing history if you use Chrome. I do not think it would take much to guess someone is thinking about having a family based on his or her search history alone.
Let's assume that based on the above, Google lowers the "pregnancy card" trigger threshold. This means they look closer at changes that might suggest your pregnant. I am not a doctor, so bear with me while I think out loud. Other than urine or blood samples, what other quantitative data is there that you might be pregnant?
For context, I recently learned that eating something that you have an intolerance to can show an elevated heart rate for two hours after eating. One test to check for allergies is to track your heart rate throughout the day. This was where this idea came from in the first place. Using a smartwatch with a heart rate monitor, plus My Fitness Pal, Google could make suggestions that you are allergic to foods you never thought of due to recognising patterns in elevated heart rate after your meals. This made me wonder what else could be possible. There's a ton of tech for tracking:
Could Google make a guess from this data alone? I cannot stress enough about my lack of medical qualifications, but I wonder if pregnancy impacts things like REM and deep sleep changes, significant blood pressure or heart rate changes at certain times of the day. Who knows, and maybe one of these things alone wouldn't be enough to know for sure, but combined, I think it will not be long before pregnancy prediction or similar could be done.
Enough about pregnancy, (Google probably thinks I am looking to start a family) what else? What things using heart rate alone could Google diagnose or push to us in Google Now? Could they push notifications to people who are diabetic to remember to take insulin? Could they diagnose diabetes? Could they flag heart problems before it is too late? I have no idea, but I'm excited to see where things go in the next few years.
Pretty funny.. my non techie friend said last night that Google knows what he is searching for before he does.
Its getting time to pull the plug on G before it is too late, just saying.........
Hmm someone sent me a link to this article
Why Google Is the New Evil Empire << google that title to read article
Then things got even creepier when Schmidt said, “We don’t need you to type at all because we know where you are. We know where you’ve been. We can more or less guess what you’re thinking about.” Then he paused and asked, “Is that over the line?”
Clearly, this creepy virtual line of Google’s is not just a moving target but a highly subjective one. How do they know when they’ve crossed it? Perhaps the more appropriate question to ask is how many lines does Google have to cross before its executives realize – before we realize – that they’re doing evil?
Whether this is evil or not depends on Google's motivation and what they intend to do with the data they collect. My ultimate personal assistant would know what I want/need without a specific prompt; there is clearly demand for this as it will increase efficiency, amongst many other benefits. It would also be great if my PA could alert me to subtle changes in my body which might be a precursor to a disease. A current barrier to the early diagnosis of cancer is the sometimes late presentation of patients to their doctor. Wearable tech may be able to detect tiny changes in biochemistry and physiology and alert the user at a much earlier stage, increasing the chance of a cure (I am a doctor!).
@craig this is great stuff! And I fully agree that this is the direction Google is going. Adding a layer of context to our lives. I have suspected for a while that the future of optimization has nothing to do with the internet at all, but everything to do with optimizing the user’s experience and what we do in our day to day lives.
SEO = Search Experience Optimization... as I say since quite many years :-)
Nice post!!! I love this kind of articles
Hi Craig,
Really insightful post; I completely agree that Google want to become the ultimate personal assistant!
You can also see that many of the start-ups in Silicon Valley are healthcare-based now, whereas ten years ago they were all consumer-orientated. This is primarily because of the fact that there's huge untapped potential in this sector. The healthcare industry is so far behind what Google is capable of doing - in fact, I've also read that Google can predict flu outbreaks if a lot of people in one area are searching for symptoms at the same time.
I was thinking about the flu outbreaks when I was reading this as well (read more on Flu Outbreaks). It would be interesting how Google will leverage this data (users and sum of users).
Maybe Google is tapping into Target's Data.
Is Google connecting data between family members yet? I imagine the divorce rate would go up, but I just need Google to remind my husband to get coffee at the store without me having to remember it myself. (Is Google slowly replacing my brain.)
Laura, Google will remember the things like a Personal Assistance for the users/you in near future.
Coincidentally, I came across a short write-up in the NY Times today, which references this article:
https://www.bmj.com/content/350/bmj.h1887
Some health professionals see the dawn of health apps as helpful, and others think it will create more paranoia and neurotic behavior than health benefits.
To end on a high note, happy mobile-geddon eve, y'all ;)
I can see them trying to do it - but it also being incredibly annoying especially when you are constantly getting notifications for things when you sort of just want to do it yourself. PA's are great, but not all the time.
It's cool that these things are possible with google, but just think about how much (more) data they will get on you... and image what would happen if that data would come into the wrong hands.
Hi Craig!
I agree with what you said in this article.
Google're surfing the Internet, we leave traces of our habits, our tastes, hobbies, etc. Making an analysis of all this, you can get to know perfectly how we are and what we want to do or we are interested in at the moment.
With Google Now they will be able to answer requests or orders, and will we be able to make suggestions that will become closer to our tastes.
The truth is that it is a very interesting topic.
We'll see how this continues to advance.
Good job!
Veeerrrry interesting! But not at all surprising. Google reminds me of the government -- they have so much technology in the works behind the scenes, stuff that could completely bend the average mind, and so they have to release it slowly--insideously--so as not to create a mass panic! ;-)
I am curious to see how the privacy issues / personal assistant sides balance out.
well its new thing to know, and it is good for webmasters, at least they will get a chance to correct themself.
Nice post. I may not fully agree with your point of view but lets see. No one can predict future.
Veeerrrry interesting! But not at all surprising. Google reminds me of the government -- they have so much technology in the works behind the scenes, stuff that could completely bend the average mind, and so they have to release it slowly--insideously--so as not to create a mass panic! ;-)
I am curious to see how the privacy issues / personal assistant sides balance out.
This is the future :)
Fun seeing what IFTTT can do. The ways it can be used are countless.
Also, with all the data that Google could collect, and some algos, we are looking at human life optimization.
Google has always wanted to be the best personal assistant imaginable. Their whole thrust is to make the user-experience the best one possible. That, along with profit, is always foremost on its mind. Thus, Google is attempting to take its service to the next level. As a personal assistant, its goal is to tell us what to expect and bring things to our attention we may not thought to consider. As it stands, Google knows your entire search history, especially if you use Chrome. So, much of the information that’s discovered in each of your search queries is based on where you’ve journeyed in the past. You might say that Google assumes you’re pregnant, and anticipates everything from diagnosis to delivery.
An article from the title promised fun and also to confirm these expectations provides useful information. Well done!
The only problem I have is that has made me realize that Google has enormous control over our lives. Shortly do a search on Google to find out how she goes to that person so long that we see instead of calling by phone ago. Get the information you need more quickly and accurately ...
I keep going back to 'the internet of things' idea when reading this. In the birth of HTML5 I was quick to predict that this would happen. I envisioned walking down the street and based on your geolocation your phone would make suggestions and provide great offers for you within your surroundings. Only makes sense really and if local businesses had a way to truly tap into that potential the world as we know it would change. This is already happening in many different ways though at the moment it seems a bit spread out as the ideas are slowly evolving.
Here's what I'd love to see though my mind can only reach so far since I'm first a software developer over a hardware engineer. I'm beginning to see robotics and electronics come together more than ever. Yes we have all of these apple monitoring gadgets, heart rate monitors etc...and that's great as they are connected to the masses, but where's the flip side of the API? Where do people start truly branching out and developing practical electronics as openly as we do software.
I know this is a trend on the rise and I've been very interested in it. Where I still get locked up is the word 'practical'. Even reading this article my mind is asking 'How can we easily monitor our eating patterns so this data can make sense'. 'How can we integrate technology in our daily lives so to truly create the most efficient 'internet of things' model.
Hello Craig, Great post about personal assistance by Google. “Find my phone”, “News for you” and many more yet to come..!!
Sounds like Google will eventually become self-aware and initiate a nuclear holocaust of mankind.
Hi criag
when world development fast and everyday, We must expect everything.
who can imagine manage the money at the mobile, or sell and buy on internet.
I think Google is on its way to be truly assistant, as things are moving and getting develop by the technology giant. That time may not be far, when visitor may be getting every to-do list from search engines.
Filling up fuel through credit card (having email notifications on transactions) and based on mileage data of your car and current fuel price, Google Now can remind you to fill up the fuel tank while passing through fuel-station.
Am I reading and imaging too much? :)
@craig this is a great article! I'm fascinated by the role of Dr Google and Google's capacity to influence something as personal as healthcare. It really does add another layer of complexity to the idea of user experience.
@Oldest_SEO is on to something IMO.
This post shows a generally positive side to this growth. However, what happens when this technology is used to help businesses make more money off of consumers by better timing push notifications, ads, etc. based on our "state of mind" (which can to some degree be determined by physiological changes + other context signals)? What happens if we go a step further and the devices we carry begin to actually interact with our physiology in various ways? Many may respond and say by the time we get there we start to seriously violate privacy, but the fact is, the general population and many in the next generation coming up have all of this baked into their heads as "ok", and we've clearly shown that if you boil the frog slow enough, he won't jump out of the pot.
Frankly, I'm not sure humans in developed countries need to get lazier, or be capable of more "output". The light side says humanity will use this type of tech to do good things. But, the fact is, at best we usually end up with a 50/50 split of good things and bad things coming out of technology that can be used in multiple directions. I personally believe, the bad, has the potential to outweigh the good here and giving Google access to more information to utilize to make money doesn't seem like a great idea long term.
Cool stuff, but I hope the implications of this are seen very clearly.
Hey Craig,
First of all I just loved the post's title, it's really cool :)
Coming to the post, I believe you're absolutely right. The latest Google feature "Find my phone" is another example of how Google is simplifying things and adding values to human's life.
Google Knowledge graph.
very funny now we can say Google make you Pregnant as well.
Ha. A bit of a false lead there, Craig. Unless you've got some unexpected plumbing, anyhow.
You're absolutely right that Google will probably be able to guess these sorts of things before us (our doctors) before long, you know, assuming a terrifying level of access to our data. Which is fair-to-likely possible.
That said, there are a number of things pure data crunching is not very good at, and one is intuitively breaking down near-infinite options. Our brains can only function because they are absolute bawlers at completely disregarding the unimportant stuff. Computers down really do that. If you need a real-world example, consider predictions of Tertiary and Quaternary (3D/Interacting) structures of proteins: https://people.math.sc.edu/dix/fold.pdf
There are so many different potential 3D end-products of even a single strand of amino acids that predictions are still way out there--unless you throw humans into the mix to rule out the ones that just intuitively don't work. A lot of medicine is the same way. It's three parts data, one part inference based on data which is not yet there; I suspect that Google will be great at telling you you're pregnant, yes, but also pretty good at telling you you're pregnant when you're not.
I just wish the government would allow 23andMe to include their health recommendations again and then add THAT layer of genetic-based data on top of everything Google is already doing PLUS all of the untapped technologies that you mentioned in this article.
Then Google could tell you that you're pregnant, have a _% chance of having a kid with brown eyes and black hair, with a _% chance of this or that disease if you continue eating / exercising the same way throughout your pregnancy. Now THAT would be the future.
Google can become your assistant in some cases only by analyzing your search history. E.g. if you search for HTML courses and spend time there, then start searching for CSS courses, after some time Google can start suggest you JavaScript courses. It's logical next step.
Or by analyzing search log. If many people are searching from mobile phones for alternative routes, Google can anticipate there are problems with traffic in that area and suggest a different route to drivers going into that area .
Google assistant can be very useful. As long as it remain an assistant, not trying to become your master, influencing (and controlling) every step in your life.
It's funny years back I asked the question 'What's stopping facebook from monitoring your conversations and catching keywords to influence their direct marketing campaigns' - I envisioned a underlying system where after analyzing the general topic the side ads would shift into a related ad campaign. Sure enough it wasn't long until Facebook did indeed try this. The general public eventually caught on and there was an outcry for protection of information and privacy.
I think there's a line to be drawn here as well. It will be interesting to see how far it stretches because right now there really seems to be very little borders to how each and every persons data is used in the name of making money.
In a simple manner, google is becoming more smart rather than personal assistance. They knows how to be helpful for searchers and people would love google more than other search engines.
I appreciate the way google tells that you are pragnent, but I like the way @Craig explained it. Because it doesn't matters what you say, but it does matters how you make feel it to people.
Great explanation mate, keep it up!