Yup, Google is listening to your audio too.
Google employees are systematically listening to audio files recorded by Google Home smart speakers and the Google Assistant smartphone app. That’s what me and my team at VRT NWS discovered in the summer of 2019.
Throughout the world – so also in Belgium and the Netherlands – people at Google listen to these audio files to improve Google’s search engine. VRT NWS was able to listen to more than a thousand recordings. Most of these recordings were made consciously, but Google also listens to conversations that should never have been recorded, some of which contain sensitive information.
You can watch the video here, English subtitles are available:
Since Google launched its Google Home device in 2016, millions of people have installed one of these smart speaker devices at home or in their office. You can ask it all sorts of trivial questions.
If you ask ‘Okay Google, how’s the traffic heading towards Brussels?’, the device automatically switches on and a computer voice will read out the latest traffic information. Seems very useful, right? You can do the same thing with the Google Assistant app on your smartphone, comparable to Apple’s Siri.
Everything is recorded
Not everyone is aware of the fact that everything you say to your Google smart speakers and your Google Assistant is being recorded and stored. But that is clearly stated in Google’s terms and conditions. And what people are certainly not aware of, simply because Google doesn’t mention it in its terms and conditions, is that Google employees can listen to excerpts from those recordings.
Google has continually claimed that it doesn’t eavesdrop. Google Holland even made a smooth YouTube ‘explainer’ to remove any misconceptions about eavesdropping. In this video, Google employees answer the question ‘Does Google eavesdrop?’. They say that the commands are being stored and transferred to Google for analysis. And they very clearly state: ‘No, you are not being eavesdropped’.
Google does listen in
It is true that Google does not eavesdrop directly, but VRT NWS discovered that it is listening in. Or rather: that it lets people listen in. We let ordinary Flemish people hear some of their own recordings. ‘This is undeniably my own voice’, says one man, clearly surprised.
A couple from Waasmunster immediately recognise the voice of their son and their grandchild.
What did we do? VRT NWS was able to listen to more than a thousand excerpts recorded via Google Assistant. In these recordings we could clearly hear addresses and other sensitive information. This made it easy for us to find the people involved and confront them with the audio recordings.
Why is Google listening in?
In the aftermath of the Amazon case, we were able to talk to someone who works for a Google subcontractor. He let us take a look at the system that collects audio via Google Assistant. Thousands of employees worldwide use this system to listen to audio excerpts. In Flanders and Holland, around a dozen people listen to Dutch recordings.
Why is Google storing these recordings and why does it have employees listening to them? They are not interested in what you are saying, but the way you are saying it. Google’s computer system consists of smart, self-learning algorithms. And in order to understand the subtle differences and characteristics of the Dutch language, it still needs to learn a lot.
Sometimes Google’s search engine has difficulty analysing a certain speech command. When that is the case, they put this command in Google’s online tool Crowdsource. By the way, if you want to help Google get better at describing images and facial expressions, everyone can use this tool (for free).
What you will not find in this free tool, is the part where you can describe audio recordings. Google subcontracts that task to outside professionals. They log in to a secure part of the tool that has a list of the audio excerpts they have to analyse. We have three sources confirming that this is the way Google works.
Speech recognition automatically generates a script of the recordings. Employees then have to double check to describe the excerpt as accurately as possible: is it a woman’s voice, a man’s voice or a child? What do they say? They write out every cough and every audible comma. These descriptions are constantly improving Google’s search engines, which results in better reactions to commands. One of our sources explains how this works.
Most recordings made via Google Home smart speakers are very clear. Recordings made with Google Assistant, the smartphone app, are of telephone quality. But the sound is not distorted in any way.
Anonymous?
Knowing that people who work for Google indirectly are listening to such recordings raises questions about privacy. In order to avoid excerpts being automatically linked to a user, they are disconnected from the user’s information. They delete the user name and replace it with an anonymous serial number. But, as shown in one of the videos above, it doesn’t take a rocket scientist to recover someone’s identity; you simply have to listen carefully to what is being said.
What’s more, if they don’t know how it is written, these employees have to look up every word, address, personal name or company name on Google or on Facebook. In that way, they often soon discover the identity of the person speaking.
Sex...
VRT NWS listened to more than a thousand excerpts, 153 of which were conversations that should never have been recorded and during which the command ‘Okay Google’ was clearly not given.
But as soon as someone in the vicinity utters a word that sounds a bit like ’Okay Google’, Google Home starts to record.
This means that a lot of conversations are recorded unintentionally: bedroom conversations, conversations between parents and their children, but also blazing rows and professional phone calls containing lots of private information.
Mistaken recordings can also occur when someone presses the wrong button on his or her phone or unintentionally gives a command.
... and violence
One of our three independent sources says he had to describe a recording where he heard a woman who was in definite distress. What are employees supposed to do with such information? We are told that there are no clear guidelines regarding such cases. It is, however, an important ethical matter. Employees only receive specific directions when it comes to account numbers and passwords. Those are marked as sensitive information.
The people that analyse these recordings also overhear lots of medical questions. You wouldn’t be the first to consult Doctor Google when you have some sort of illness.
The recordings also strikingly confirm one of the Internet’s rules: men seem to look for porn a lot, even via smart speakers.
Professional confidentiality
When we confront him with our findings, Bavo Van den Heuvel, a cyber security expert, says: ‘This is shocking.’ He immediately points out the possible dangers: these recordings can be made anywhere, in a doctor’s surgery, for example, or in a police or judicial context, where people are dealing with sensitive, private matters.
Google reacts: "This work is crucial to develop new technologies"
In a reaction on the VRT's report, Google admits that it works with language experts worldwide to improve speech technology. "This happens by making transcripts of of a small number of audio files", Google's spokesman for Belgium says. He adds that "this work is of crucial importance to develop technologies sustaining products such as the Google Assistant." Google states that their language experts only judge "about 0.2 percent of all audio fragments". These are not linked to any personal or identifiable information, the company adds.
If you have a Google Home speaker and don’t want to be recorded unintentionally, read this: How to prevent Google eavesdropping (in Dutch).
Google is not the only company that works this way. In April, the Bloomberg news agency revealed that American internet giant Amazon also does it. Bloomberg also had evidence that, just like Google, Apple subcontracted people to train its well-known Siri search assistant.
- story by Denny Baert, Lente Van Hee, Ruben Van Den Heuvel and Tim Verheyden