Virtual Tech Tools Fuel Higher Ed Robotics Research

“We’ve got the digital twin of a humanoid robot,” Jamerson says. “You could run the simulation and be in a situation where a robot arrives at an oil rig, there’s an emergency, and it doesn’t know what to do. But within five minutes, you could train it on a week’s worth of trying different approaches.”

VR Amenities Support Collaborative Robotic Research Efforts

In recent years, computing power and VR advancements have facilitated broader possibilities for robotics research and use, says Nancy Amato, IEEE Robotics and Automation Society president and director of the University of Illinois at Urbana-Champaign’s Siebel School of Computing and Data Science. 

“Before, robots in factories would typically be in cages to keep people away from them for safety reasons,” Amato says. “We are ready to develop systems that can be aware of their surroundings. You can have a virtual headset and ask the robot to implement instructions. That can help someone learn how robot operation works, and how to program the robot.”

Virtual research experiments can also familiarize college students with robots’ tech components, ranging from computing systems that process interaction data to sensory elements such as cameras. These are crucial inclusions, Jamerson says, because robots are generally much stronger than humans and could easily injure them.

DISCOVER: Virtual reality and artificial intelligence can help students improve soft skills.

The connectivity methods used to convey information can vary depending on research project perimeters. 

Although some of Jamerson’s work, for instance, has employed Wi-Fi and Bluetooth, when NASA asked his team to create a digital twin of a facility that uses robots in production four years ago, the federal agency’s security requirements prohibited wireless device use. 

The team created a LIDAR scan of the rocket factory, built software for onsite and remote employees to work collaboratively with robots located in the building using Apple iPhone devices and Meta Quest 3 VR headsets. Then, researchers designed and produced wired sensors, which Jamerson says gave students a chance to learn about physical computing and digital fabrication.

“We were working for a major government partner, and for our subsequent work, we’ve had to build a rather unique university framework that includes being able to work with controlled unclassified information,” he says. “Everything served out to the user was via AWS GovCloud.”

XR Studio’s space also sports amenities such as gaming computers with powerful GPU processors, a virtual film production studio, NVIDIA DGX Spark AI supercomputers and a robotic arm. LSU built a Cave Automatic Virtual Environment in 2017 with 55-inch display screens.

“The entire campus is tied together with a very high-speed network,” Jamerson says. “We benefit from a wonderful partnership with NVIDIA that provides our GPU clusters. We have a LED wall here that’s 25 feet by 12 feet that allows us to make a quite a large window into the digital world.”

Universities that hope to integrate VR in robotics research might need to add computers to run robot technology and virtual environments simultaneously, Amato says.

“There's a lot of edge computing in research,” she says. “When you want to do something in the moment, you have to be very careful about latency and consider how much you can do remotely in the cloud.”

LEARN MORE: Different types of virtual environments have different benefits.

Multiple Human-Robot Collaboration Approaches With Various Tech

A number of VR-focused research initiatives center on nonverbal communication — almost subliminal physical signals that indicate people’s intent, according to Mark Gross, computer science professor and director of the ATLAS Institute at the University of Colorado Boulder. 

“You get robots explicitly programmed to recognize the tells, or look at lots of data sets of people doing things together, then extract what those tells are,” Gross says.

The ATLAS Institute provides an array of tech resources, including Apple Vision Pro and Microsoft HoloLens mixed-reality headsets, for student robotics projects. 

Master’s and doctoral candidates in Tufts University’s Human-Robot Interaction program receive mathematical and computational foundation instruction and can take electives such as a psychology course on human communication.

“They have to understand both the technical side of the robotics — what a robotic architecture is, what systems there are, how you get robots to do stuff — but also how you would evaluate how well a robot works that interacts with people,” says HRI Laboratory Director Matthias Scheutz. “So, they get some experimental design and evaluation methodology as well.”

Scheutz has used eye tracking functionality and HTC VIVE virtual reality headsets to measure pupil diameter changes and identify the cognitive load tasks require. Participants sit in a rotating chair and may interact with robots and humans in different rooms.

“We can build software systems that allow robots to adjust their behavior based on how busy people are,” he says. “If a person is cognitively overloaded, the robot can try to proactively help them. The VR turned out to be a really good research tool that augments the work we do with physical robots.”

Augmented reality, which allows for video pass-through during experiences, can also be beneficial, according to Ellen Do, director of UC Boulder’s ACME Lab, which explores computational tools for design, cognition and embedded interaction.

UP NEXT: Virtual reality can bridge the skills gap in higher ed.

“With VR, you have a headset; you don’t see anything else,” Do says. “But augmented reality is a mixed reality. You can still see your physical environment. You have the Google map telling you which way to go to a certain landmark, and as you walk, the virtual arrow could point out which way to turn.”

Jivko Sinapov, an associate computer science and mechanical engineering professor at Tufts, began using AR in robot research because two undergraduate students who were fans of the Pokémon GO game suggested adding it. 

The technology has been useful in the debugging process, Sinapov says

“Typically, the robot data is visualized on a computer screen,” he says. “You constantly have to turn your head from the robot to the screen. With AR, we were able to see what is happening with the robot and its different types of sensor data directly overlaid on the physical world — and can recognize when, for example, there’s a failure.”