This article was originally published on March 13, 2012.
Augmented reality is a technique used to superimpose computer-generated images for visual enhancement of live images. Augmented reality can also be characterized as dynamic overlays when computer-generated images are registered with moving objects in a live image.
While the better-known concept of ‘virtual reality‘ (VR) concerns immersive artificial worlds, ‘augmented reality’ (AR) involves blending the real world with simulated elements.
The technology has already been employed in niche areas such as flight simulations and surgical training, but AR’s range of uses has increased as available computing power has grown.
“If pilots are not familiar with the airport, they have to stop and pull out maps,” said Trey Arthur, an electronics engineer at NASA Langley Research Center in Virginia. “This display, in the new world where these routes are going to be digital, can tell them what way they’re on, where they need to go, where they’re headed, and how well they’re tracking the runway’s center line.”
Similar heads-up display (HUD) technology exists for military fighter jet and helicopter pilots, who wear helmets with the transparent displays. Newer commercial airliners also use HUDs installed in the cockpit, but lack the head-tracking “augmented reality” of NASA’s technology that layers virtual images or maps on top of a pilot’s real-world vision.
Giving pilots better awareness on airport runways swarming with airplanes could prevent catastrophes such as the world’s deadliest aviation incident at Tenerife in the Canary Islands in 1977.
A collision there between a jumbo jet trying to take off and another jet sitting on the foggy runway ended in a blazing inferno that killed 583 passengers and crew.
Arthur and his NASA colleagues made their head-worn display by combining a Rockwell Collins helmet display for soldiers with their own head tracker custom-built by Intersense.
The head tracker combines built-in gyroscopes and a camera that detects passive paper targets pasted around a cockpit to tell where a pilot has turned his or her head.
The head tracker combines built-in gyroscopes and a camera that detects passive paper targets pasted around a cockpit to tell where a pilot has turned his or her head.
The ESA designed Wearable Augmented Reality (WEAR) is a wearable computer system that incorporates a head-mounted display over one eye to superimpose 3D graphics and data onto its wearer’s field of view.
Controlled by voice for hands-free operation, WEAR includes onboard location and object identification to show astronauts precise information about what they are looking at, as well as providing step-by-step instructions to guide them through difficult, lengthy procedures.
“At the moment, International Space Station (ISS) crews still use paper instruction manuals for many operational and maintenance tasks,” explains Luis Arguello of ESA’s Modelling and Simulation Section, overseeing the WEAR project. “Obviously, it’s easier to perform a task while holding instructions in your hand. So we have developed a new type of user interface that is easier still, allowing astronauts to be guided precisely in their work without holding anything at all.”
Key hardware elements include a mobile computer connected to a headset with a head-mounted display, a pair of video cameras – for bar code reading and objection recognition – and an Inertial Measurement Unit (IMU).
The image-recognition system allows WEAR to know where its user is located, by checking his viewpoint against stored 3D information about the module, derived from a Columbus Computer Aided Design (CAD) model.
On the software side, WEAR incorporates a 3D AR toolkit as well as speech recognition and synthesis technology, object recognition and tracking systems and commercial bar code reading technology. Reading the bar code allows the quick identification and retrieval of information of ISS items stored in the onboard Inventory Management System.
The goggles can track a pilot’s head-movements and overlay runways, towers and potentially other airplanes over their view – an invaluable tool when fog rolls down across an airport.
Even as the pilot turns his head, the goggles can react in real-time to ensure the virtual representations always stay in the same place as their real-world counterparts.
