NASA is building the next Mars rover in mixed reality

I poked my head inside NASA's latest Mars rover. During a mixed-reality demonstration at the Jet Propulsion Laboratory (JPL) in Pasadena, as soon as I strapped on a HoloLens, a vibrant holographic vehicle popped up in front of me.

I walked around a model of the next-gen rover, which was only visible to the headset wearers in the room, before crouching down for close inspection. As I leaned in to look at one of the corners, the spacecraft's platinum-gray exterior disappeared to reveal a set of neatly aligned internal parts. As I continued to inch forward, my movement kept peeling away the layers of the rover until I could see the floor beneath my feet again.

The ability to investigate the design of NASA's latest spacecraft, which is being engineered for the next mission to Mars in 2020, makes ProtoSpace a unique use for Microsoft's augmented-reality headset. The project enables JPL designers and engineers to not only visualize their designs as virtual 3-D objects, but also pinpoint potential problems that could affect space missions along the way.

ProtoSpace is one of two mixed reality experiments at JPL. With the other, more widely known application, OnSight, the team has been using the new visual medium to recreate a hologram of Mars based on the Curiosity rover's current exploration. Strapping on a HoloLens can quickly transform the floor into a rusty red planet, complete with its rocks, cracks and craters.

The medium takes away some of the complexities of machines and tools to make space exploration more experiential even for scientists. "Mixed reality brings data from our missions and the environment they're exploring to us in ways that feel familiar and natural," says Jeff Norris, lead mission operations at JPL. "It [replicates] the ways in which we look at the world today. I see it as not only simplifying the process of space exploration for our explorers but also allowing them to gain insights that they wouldn't access if we weren't engaging their natural abilities as effectively."

Back in Pasadena, ProtoSpace is engaging engineers in a new kind of interaction with their spacecraft design. The vehicle, which is essentially a Computer Assisted Design (CAD) model in holographic form, appears as a floating, dynamic animation. You can do things like slip your arm through the brightly colored parts and bolts that are projected in front of you. But what the rover lacks in definition it makes up for in presence. Unlike the CAD design on a flat computer screen, mixed-reality technology brings a full-scale replica of the rover to life inside a room.

"It's very important for engineers to gain intuition for the actual size, shape and arrangement of the spacecraft," says Norris. "They need to see these designs at full human scale so they can use their natural abilities for the feeling of how large a part is or how tight the clearance might be. It's difficult to access that when you're looking at a model on a computer screen."

When working on the new model the team has been able to place metal prototypes against the exact virtual spots where they need to go. This overlay of physical objects on a holographic rendering has given the team a new way to measure distances between objects for a precise alignment of the parts. "Being able to see it in the space, walk around it and put their arms in the hologram allows engineers to uncover a lot of solutions," says Matthew Clausen, lead designer on ProtoSpace. "[They can] discover problems they didn't even know existed." While assembling the spacecraft in a "clean room," for instance, engineers can position the hologram to see how the spacecraft would align with the crane and other moving parts to avoid delays during the final build.

Spacecraft design is a complex and collaborative process. It requires meticulous planning to keep a space mission that's worth billions on track. "Our launch windows are often not very flexible," says Norris. "They're dictated by the motion of the planets. A problem that causes us to stop assembly for a while can put us a week off our schedule. We plan for those, but when too many of them occur they can threaten the launch date, and that's a huge impact."

Mixed reality is helping scientists predict and overcome obstacles in innovative ways. But as a new medium, it presents its own challenges. When designing in CAD, for instance, a designer can rotate objects or zoom in and out for context. "With a full-scale spacecraft [hologram], rotating it can be really disruptive when you have this huge thing shifting in your space," says Clausen. "Working out new modes of interaction has been fun and experimental. Every time there's a new solution, there's a new side problem that needs to be worked on."

The holographic medium presents a new way of assessing design objects in a room -- but the interaction isn't limited to real and virtual models. The ability to lean in and poke around a hologram while still being aware of other people in the room makes it an appropriate medium for designers. "It offers an opportunity for engineers to collaborate," says Norris. "Many of JPL participants are overseas and it's been a challenge to keep everyone on the same page when the physical designs and mockups are here in JPL. With ProtoSpace they can look at the full-scale model -- that's a capability we've never been able to offer before."