Going Mobile: Government and Consumer Applications of Immersive Environments

Lewis Shepherd (Reston, VA) -

While everything may be “going mobile” in the not-so-distant future, that doesn’t necessarily mean that everyone will be carrying mobile devices as we know them today. Here, I discuss a different future path from what is typically portrayed as a world of always-connected handheld mobile devices - an alternative but probable future in which people leverage digital services within a immersive environment of virtual computation.

Everything’s Going Mobile… But Do You Need a Device?

Like many people, I was very impressed by a video over the weekend of the Word Lens real-time translation app for iPhone. It struck with a viral bang, and within a few days racked up over 2 million YouTube views. What particularly made me smile was digging backwards through the Twitter stream of a key Word Lens developer whom I follow, John DeWeese, and finding this pearl of a tweet from several months ago as he was banging out the app out in my old stomping grounds — the San Francisco Bay Area. That’s a hacker mentality for you!

But one thought I had while watching the video was: Why do I need to be holding a device in front of me to benefit from its computational resources and display? While, like many others, I’ve seen the predictions that “everything’s going mobile,” most people assume that “mobile” refers to a physical device rather than a state of being.

I forsee an alternative future. This possible future is one in which people will take advantage of computing resources and digital services all around them in a supercharged immersive environment of virtual computation. This is distinct from something like Second Life, where you log-in to a virtual world divorced from your real one. Instead, immersive computation is being rapidly integrated all around us, in our everyday world, and fairly soon we won’t need to pull out a smartphone to see it.

Air Guitar Competition From the Future

What do I mean by “supercharged immersive environment of virtual computation,” anyway? Here’s what I mean: I’m pretty sure this video is exactly what everyone has always thought “playing air guitar” should be like.

This work from London-based developer Chris O’Shea represents a terrific “private-hack” application that uses Microsoft Kinect as a platform for sensing, ranging, and interacting with 3D virtual environments.

Collaboration Inside Virtual Environments

Did I say interacting “with” virtual environments? How about interacting “in” them? Observe this stunning example which highlights the potential for uses in professional collaboration and virtual-object-manipulation within immersive spaces. Unlike an environment such as Second Life, here you don’t control an avatar with a keyboard or mouse – you are the avatar.

(Note the funny hat-tip to the classic movie Office Space at the beginning of the video - Apparently, even Martian cubicle-workers have to deal with TPS reports!)

In the past month, there has been an explosion in Kinect-hack activity, AKA “OpenKinect,” with individual developers doing work on Microsoft platforms and on a host of others, including libfreenect/python, OpenFrameworks, Apple’s OS X, and even iPad.

One of my favorite OpenKinect examples is work at the Georgia Tech College of Computing. There, researchers have built the Kinect American Sign Language Recognizer. One huge behavioral advance from earlier pre-Kinect technology is that the deaf children no longer have to wear cumbersome, unnatural headgear and wrist-mounted 3D accelerometers, as you can see in the research video here.

Other creative OpenKinect examples include controlling a hands-free web browser, an artful “invisibility cloak” use, and nifty ideas highlighted in Kinect Hack Prize Competitions and this combo-video of “12 Best Kinect Hacks.”

The company’s position on these hacks has been “cautiously supportive,” one might say, based on this account from the 11/22/2010 PC World:

“Kinect was not actually hacked,” said Microsoft program manager Alex Kipman, speaking on NPR’s Science Friday with Ira Flato. “Hacking would mean that someone got to our algorithms, that sit inside of the Xbox, and was able to actually use them, which hasn’t happened. Or it means that you put a device between the sensor and the Xbox for means of cheating, which also has not happened. That’s what we call hacking, and that’s why we’ve put a ton of effort to make sure it doesn’t actually occur.”

“What has happened,” continued Kipman, “is someone wrote an open source driver for PCs that essentially opens the USB connection, which we didn’t protect [on purpose], and reads the inputs from the sensor. The sensor, again as I talked earlier, has eyes and ears, and that’s a whole bunch of noise that someone needs to take and turn into signal.”

Microsoft Game Studios manager Shannon Loftis weighed in as well, noting that, “as an experienced creator, I’m very excited to see that people are so inspired that it was less than a week after the Kinect came out before they had started creating and thinking about what they could do.”

Government Applications of Immersive Environments

These hacks of Kinect demonstrate that, every so often, computing takes a “fun” turn again, delighting and enticing a new generation of whiz-kid programmers and developers. But while having fun with computing is amazing in itself, there are also more serious applications of such creativity and innovation to the public sector.

Developers in the government space are getting quite interested in Kinect and other immersive technologies, and I’m fascinated with the possibilities already emerging on white-boards or dev-laptops in areas like education and government training, health-care applications, and easier and more user-friendly government/citizen interactivity in general.

In the mainly civilian realm of work, there have been two great summaries so far: My colleague Chris Niehaus, Director of Microsoft’s U.S. Public Sector Innovation, has explored many of the state-of-the-art possibilities and social implications in health and medical care in a piece for SECTOR:PUBLIC, and Alex Howard has written about other non-gaming areas where Kinect immersion could take hold on the O’Reilly Radar blog.

As you might expect, national-security efforts in this (as most) area are kept mainly private, but we know that immersive environment possibilities are already being explored in robotics, ISR (intelligence, surveillance, reconnaissance), and military command-and-control. In one publicly-available example, members of UC-Berkeley’s Hybrid Systems Lab STARMAC Project have hacked up a Quadrotor UAV with an onboard Kinect Sensor to demonstrate the off-the-shelf quality of environment-sensing and remote control:

While researchers are still in the early stages of OpenKinect and related projects, the enormous potential of such relatively inexpensive consumer technologies combined with some experimental gusto has rapidly become clear to both the private and public sectors.

First It’s Weird, Then It’s Scary, Then It’s Normal

Alongside commercial uses of human-computer-interaction (HCI) and natural user interfaces (NUI), the future will definitely feature incredibly powerful government applications of these technologies. And the initial version of Kinect will eventually be surpassed with more advanced iterations, and likely compete in a healthy market of alternative hardware enabling depth-sensing and touch-free interactivity.

When that happens, people at home and at work will increasingly see and use what I call “Air Everything” (or almost everything). And we’ll like it. And then in many circumstances we’ll forget we’re even using room-based or location-embedded computing resources.

Here’s an analogy from a century ago: In the early days of home electricity, there was enormous awareness – and fear – of the power coursing through wires behind floorboards and across walls. Early electrical plug-in outlets were frightening objects and the source of great parental anxiety and dangerous childhood experimentation.

But eventually, with new technology integrated into home design and with buried power lines and power-grids incorporated into urban architecture, we lost sight (literally) of the electricity being transmitted all around us. Now, things just “turn on.” NUI will become that carefree. And device-free.

Tracking the Kinect Hacking Universe

To keep up with the frenetic pace of activity around the Kinect platform, check these sources periodically: KinectHacks.net and its frequent updates; and the YouTube channel of UC-Davis computer science professor and Kinect sensei Oliver Kreylos, whose videos are a mix of eye-popping functionality and behind-the-scenes programming explanations.

Lewis Shepherd is Director of the Microsoft Institute for Advanced Research in Governments, part of Microsoft Research. A version of this post appeared as “Air Everything“ on his blog Shepherd’s Pi. Lewis previously wrote “Putting the Human in Computing” for SECTOR:PUBLIC.

Images of air guitar, Kinect girl and light switch and green/red outlets used under Creative Commons.