Telerobotics: Unifying Human and Robotic Spaceflight

On Monday, September 7—Labor Day holiday, in the United States—a little blue rover the size of an all-terrain vehicle crossed a room at the European Space Research and Technical Center in Noordwijk, South Holland. The rover, named Insight Centaur, was under the remote control of ESA astronaut Andreas Mogensen, who floated in the International Space Station’s Columbus module as the orbiting complex sped around the Earth.

Using video displays and a vibrating, force feedback joystick—which allowed the astronaut to "feel" what Centaur's robotic arms felt—Mogensen guided Interact over to a table and plugged in a small test peg. All told, it took him 45 minutes to get a feel for the rover's controls and finish the job. On a second try, his time improved to less than 10 minutes. 

It can take days of careful planning and execution for ground-based operators at NASA's Jet Propulsion Laboratory in Pasadena, California to carry out similar precision tasks with the Curiosity rover on Mars. That's partly because it takes anywhere from four to 24 minutes for radio signals moving at the speed of light to travel between Earth and Mars, depending on the planets' positions. 

Mogensen's time delay, on the other hand, was a mere half-second—even though the rover's data stream was traveling from the Netherlands across the Atlantic Ocean, to Mission Control in Houston, over to a ground station in New Mexico, up to NASA's Tracking and Data Relay Satellite System, and on to the ISS. In theory, a human in Mars orbit could operate Curiosity in near-real time. 

To be sure, there are challenges in giving an astronaut the ability to make split-second decisions normally vetted by entire teams. But if applied correctly, telerobotics—with humans nearby—just might be the perfect unification of human and robotic spaceflight. Two groups in particular at NASA and the European Space Agency are working to make this fledgling technology commonplace. And they're using the International Space Station as their testbed.

The ESA approach

The Europeans make good use of their time aboard the station. Mogensen, part of a three-person team that delivered a fresh Soyuz spacecraft to one-year crewmembers Scott Kelly and Mikhail Kornienko, was only at the orbiting laboratory for seven days. Yet the Danish astronaut successfully completed 15 separate experiments—including driving two different rovers.

Philippe Schoonejans is the project manager for Meteron, the Multi-Purpose End-To-End Robotic Operation Network, which sponsored Mogensen's telerobotics experiment. "The one-week stays are completely packed with content—more than what you have for people that are staying half a year on the ISS," Schoonejans told me. "No astronaut and ground teams could work with that intensity for a full half-year. We have had lots of people working in the months preceding this mission until very, very late in the evening."

NASA and ESA collaborate on space station telerobotics through an unfunded agreement that covers everything from use of NASA's Tracking Data and Relay Satellite System to coming up with international standards for telerobotics communications. Meteron began three years ago, when NASA astronaut Sunita Williams controlled a Lego robot from the ISS. In June, NASA's Terry Virts shook virtual hands with telerobotics specialist André Schiele, who was the principal investigator for Mogensen's experiments. This December, another ESA astronaut, Tim Peake, will launch to the station for a six-month stay that has more rover driving on the schedule.

ESA telerobotics favor low-latency, video-heavy data streams that employ clever tricks like force feedback joysticks. Schoonejans said one of the biggest challenges is funneling so much data up to the station over a 1-megabit-per-second data connection (my home Internet connection has an upload speed about 12 times faster). ISS data links were designed so that astronauts could send lots of engineering and experiment data back to the ground; not so much in the other direction.

"We used a new communications method called DTN—Disruption Tolerant Networking. It's sometimes marketed as being 'space Internet,' " Schoonejans said. One of the features of DTN is that data can flow to its destination through multiple paths, and switch gracefully when one path becomes unavailable. That's handy for a space laboratory traveling in and out of range of satellites and ground stations around the Earth, or for a spacecraft millions of kilometers away in Mars orbit.

One Mogensen rover experiment involved five simultaneous video streams. He used a rover called Eurobot to remove a payload from a lander, all while a small, secondary rover observed the scene. During early tests, the video was up to a minute late and out of sync. But engineers worked out the kinks, and by the time Mogensen tried the experiment from orbit, he completed the task much faster than ground controllers expected. "That was quite challenging, in terms of timing, to make sure that all of these things reach their destination at a similar time," Schoonejans said.

The NASA approach

Both ESA and NASA are testing telerobotics that give astronauts so-called "supervisory" control, where the operator sends the robot a set of preprogrammed commands and then monitors in near-real time. If the robot gets into trouble, the human can quickly take manual control, said Terry Fong, a NASA project manager for human exploration telerobotics. I reached Fong by phone at the NASA Ames Research Center in Moffett Field, California. 

"Because the delay is short—in our testing it was only a second, round trip—even though you're commanding at a supervisory level, you can monitor real-time," Fong said. "You can still interactively watch what's going on. And then when the robot has problems, you can recognize that right away, and switch modes of operation."

NASA telerobotics favors this approach. It's the default mode of operation for Mars rovers—minus the real-time human intervention. When mission planners want to send Curiosity or Opportunity from point A to point B, they can also give the rovers the ability to evaluate obstacles in their paths and potentially get themselves out of a jam.

On an outdoor, rock-strewn landscape at NASA Ames, ISS-bound astronauts have completed three tests of a rover called K10. The first was completed in June 2013, when NASA's Chris Cassidy commanded K10 to unroll strips of kapton film to build part of a mock radio telescope that could be deployed on the far side of the moon.  ESA's Luca Parmitano and Karen Nyberg of NASA completed subsequent deployments and inspections.

"Overall, it was a nice set of experiments," Fong said. His team's original plan was to observe how each astronaut responded to the rover control interface, and monitor for individual differences. But since Cassidy, Parmitano and Nyberg were all aboard the station together during Expedition 36, they collaborated more than Fong expected. 

"One thing we did not foresee was how the crew would talk to each other," he said. "At one point during a session we saw a hand floating in the video frame, and we're like, 'Who's hand is that?' We realized, 'oh, it's one of the other crew, in real-time, giving advice,' and we had not planned for that." 

Fong is now working with a rover called K-Rex to support a proposed NASA rover called Resource Prospector, which would launch in 2020 to mine hydrogen, oxygen and water from a lunar polar region. When engineers started investigating real-time rover control for the mission, they realized NASA's Deep Space Network—specifically, computer algorithms used to prevent data loss and correctly order incoming space packets—might not be up for the job. 

"Because of the inherent buffering in the DSN, the communications delay from a robot on the surface of the moon—direct line-of-sight to Earth—is not a 1.2-second light speed delay. It's more in the 15-to-20-second range," Fong said.

But a human in lunar orbit at just the right time—perhaps when the rover is focused on precision tasks requiring a lot of human intervention, as opposed to driving long distances—might be able to remote-control the robot in ways that aren't possible from Earth. This could especially come in handy for rovers with shorter lifetimes, or ones that must hibernate during the lunar night.

"A highly available, high-bandwidth link for short periods of time when you're setting things up could be very meaningful," Fong said.

On to Mars

NASA plans to begin sending humans to cis-lunar space as early as 2021, using the moon as a proving ground before heading to Mars in the 2030s. ESA is watching carefully to see what role the Europeans can play, and telerobotic science missions could be a part of that. 

"I think scientists are very excited about the moon, too," Schoonejans said. "There are maybe different opinions, depending on who you ask, about whether the moon is purely a proving ground for Mars, or whether it is by itself scientifically interesting. I think for us, it is both."

Schoonejans said ESA is defining telerobotic mission scenarios that fit with NASA's SLS and Orion plans. He hopes to secure funding to continue developing concepts at an ESA ministerial conference next year.

Fong is also watching. He said that the ultimate role of telerobotics in NASA's Mars plans will be defined by specific mission goals, and the different ways these goals can be accomplished. 

"I think it's a little misleading to think, 'oh, just because we have a short delay, that somehow solves everything,' " he said. "I think it's more important to think, because you have humans in proximity, it gives you more options for operations than you have just from Earth."