The Dark Sides of XR

date

Jul 30, 2020

type

Post

year

slug

dark-sides-of-xr

status

Published

Oh the possibilities of Mixed Reality!

Virtual content can be added into your real environment via a device so full of sensors, cameras and microphones that it knows not just everything you say and see, but also how you feel about it.

Fantastical possibilities arise from this! We could finally have stories that are custom-tailored to our interests. There could be real-time computer-animated movies that automatically adjust the plot to our reaction. If it detects a strong dislike for a certain character, build him up to be the antagonist, then kill him. If it sees your eyes linger on that person in the background while your heart rate goes up just slightly, well, guess who just earned themself a love-scene in the story. And not just movies could benefit from this! The games you play could skip the parts that you don’t like and extend the sections you enjoy the most. The difficulty could automatically adjust to your level of frustration or enjoyment. The virtual books you read could skip or shorten the parts that you don’t care about and automatically highlight the sections that evoked the strongest reaction from you.

Virtual assistants could slowly shift the tone of their voices to find the one voice that you enjoy hearing the most.

Magic Leap’s Mica doesn’t talk at all yet. She communicates through intensive staring.

Everything you see could be analyzed and whenever you wonder about something you see, information about it could pop up right next to it. If you look at text in a language that you don’t understand, the translation could automatically be super-imposed, spoken words could be subtitled or even translated in real time.

What an interesting array of possibilities, what a fantastical new world!

Too bad we’re living in the real world where the main focus will be on manipulating you…

Manipulating you

Advertisers are already investing insane amounts of time and resources into tracking your interests and whereabouts. If you look at a product on Amazon you’ll see that exact item pop up in ads on all sorts of other websites shortly afterwards. They’ve got you on the hook, they know you want this thing, now they just need to reel you in by reminding you about it every chance they get until you give in. Now imagine how excited that industry is about these new possibilities.

How much data do Facebook, Google or Amazon already have about you? They know everything about your social life already, they know who you talk to, they know where you go, they know how often you look up your former partner on the internet, they know when you go to visit your doctor, they know what books you read, what music you listen to, what products you buy, what products you dream of buying. They have all your photos and videos, they know what you look like on a bad day, they know the entire inside of your house, they know what your mother’s voice sounds like, they know that your daughter had a rash when she was 3. They have all your scanned documents, they know how much money you earn and how much money you spend. Their virtual assistants are on your phone and in your home, always on, listening in on every word you say.

So all your data is in the hands of a few money-focused companies trying to influence your buying decisions. And that is actually the best case scenario. Because the alternative is so much worse.

I’m not exactly looking forward to product placement showing up in my childhood photographs, or being told to buy something in a voice that’s suspiciously close to my mother’s. All of this is terrible, but harmless in comparison to the alternative: What would happen if someone with less capitalistic intentions would get access to all this data about you?

The simple truth is, that they could probably get you to do anything they wanted. And without you even realizing. They would know everything about your interests - it would be easy for them to feed you content that you’d deem highly relevant and they would work their own messages into it. All they’d need to do would be to filter and tweak your news feed. They could manipulate you in very subtle ways over a long period of time and you would never even know.

And now imagine they had this additional layer of feedback where they’d instantly know your emotional reaction to every piece of information you are being fed. Not only would they know every word you’d read and every picture you look at - they would also know all your reactions. And they would use that to train a machine learning model, that would be able to predict your exact reaction to every little snippet. Then they could create a stream of information so perfectly tailored to your interests that you would gobble it up with joy, unaware that they had worked their own sub-context into every single part of it. Everything carefully constructed and placed with the sole purpose of twisting your views and pushing you in a certain direction. This technology would even enable them to adjust to your reactions in real-time and oh, how very effective this would make it!

Little harmless-seeming bits would be added to every article you read, to every book, to every social media posting of your friends. They’d be hidden in subordinate clauses and irrelevant details. Suddenly it would have been a Russian man who saved that child instead of just a man. Or a German man would have robbed the bank instead of a local man. And over time an image would begin to form in your head, stereotypes would shift and the next time you’d have to make an important decision all of this would play a part, whether you realized it or not. Over time your views could be shifted quite severely — maybe you’d come to realize that yes, freedom of the press was overrated, because all you’ve seen for months and months were stories with a negative hue towards that topic.

And you won’t even be able to trust your family and friends. Within the next few years we’ll have photo-real life-like avatars that will replicate all the little quirks and traits that make us instantly recognizable by our friends and family. We’ll be able to feel someone’s presence even though they’re half a world away. Great. But how will we know it’s really them? Facebook is working on stunningly life-like avatars. And once they have created a global library of traits and trained their models on thousands of people, it will take no more than a video to create an avatar of you. Good thing, we’ve been uploading gigabytes of videos and photos to Facebook for the last 10 years. Combine that with the likes of Lyrebird who can already do “ultra-realistic voice cloning” based on just a 1 min audio-clip of your voice and you can never be sure who you’re talking to ever again. So this doesn’t just affect shared virtual spaces, no, it affects all phone-calls, all voice-chats, all video-conferencing — it will all be unreliable because it will be trivial to recreate anyone’s exact likeness and voice. Scammers have shown repeatedly how easy it is to get someone to do something by impersonating someone they trust. A quick phone call to someone from the spoofed number of their partner, tell them you urgently, urgently need their social security number because you have to fill out a form. They trust the number on their display, the sense of urgency in your voice does the rest — you don’t even have to recreate the voice for this one — they don’t even notice if it’s a little different. Sending emails from any email address is stupidly easy. If your dad gets an email from your address asking him to unpack the attached zip file for you, would he think twice about it? I doubt he would check the metadata to see if the mail is actually coming from the mail-server of your mail-provider?

We’re almost at a point where the only communication you can truly trust will be face to face in the real world.

But I still want XR to happen!

I want life-like avatar chat, augmented reality, virtual assistants and personalized movies and games. How do we still make this work without sacrificing our freedom, our integrity and our sanity?

The worst possible outcome for us is a platform that is not open, where one company controls the hardware and the entire ecosystem. A closed hardware platform with a closed-source operating system that is controlled by a single company means that this company can monitor everything that every single user does on the platform. It means that they can decide who can and can not publish on that platform and that every bit of software on your device goes through their system and is under their control.

This is why I never liked iOS. Apple spins some yarn about security and privacy, but I don’t buy it. I feel secure and private if my data either stays on my device, under my direct control or if it’s encrypted on my device and then stored in the cloud in encrypted form.

Incidentally this is also why I was never happy at Magic Leap. Their vision of a “Magicverse” and all this talk about building an “ethical foundation” sounds nice, but believe me — investors didn’t give them billions of dollars to build an “ethical foundation”.

This is too important to leave to companies that live their lives in the stranglehold of investors and shareholders (the two most money-focused entities in existence). If a company’s CEO ranks amongst the richest people on the planet, then that should be a good indication that their main focus might not be on the common good — no matter what they say.

So how do we avoid an Orwellian dystopia and get an open, privacy-focused XR platform instead?

In all seriousness: I don’t know. Our best chance might be that they overstep and ruin it for themselves. There’s a few notable attempts of creating open source XR platforms (Project North Star!), but will they be able to compete with the capitalist force of the likes of Microsoft or Apple? As long as we’re all just focused on making money (and don’t get me wrong, we have to be if we have rent to pay and kids to feed), how could we create a platform that’s first and foremost focused on improving our quality of life rather than being engineered to generating income with the life-improvement-part being tacked on by the marketing team to make it sell better?

I want life improvement without ugly footnotes. But it’s not looking good.

The end.

ℹ️

This is a repost of a blog post from my old blog on medium