Editor’s Note: This article was originally published in 2012, highlighting early concepts for augmented reality displays in aviation and space exploration. Augmented reality technology has evolved significantly since then, but these foundational projects paved the way for modern HUD systems.
Augmented reality is a technique used to superimpose computer generated images for the visual enhancement of live images. Augmented reality can also be characterised as dynamic overlays where computer generated images are registered with moving objects in a live image.
While the better known concept of virtual reality involves immersive artificial worlds, augmented reality involves blending the real world with simulated elements. The technology has already been employed in niche areas such as flight simulations and surgical training, but its range of uses has increased as available computing power has grown.
Enhancing Pilot Awareness
“If pilots are not familiar with the airport, they have to stop and pull out maps,” said Trey Arthur, an electronics engineer at the NASA Langley Research Center in Virginia. “This display, in the new world where these routes are going to be digital, can tell them what way they are on, where they need to go, where they are headed, and how well they are tracking the runway’s centre line.”
Similar heads-up display (HUD) technology exists for military fighter jet and helicopter pilots, who wear helmets with transparent displays. Newer commercial airliners also use HUDs installed in the cockpit, but they lack the head tracking augmented reality of NASA’s technology that layers virtual images or maps on top of a pilot’s real world vision.
Giving pilots better awareness on airport runways swarming with aeroplanes could prevent catastrophes such as the world’s deadliest aviation incident at Tenerife in the Canary Islands in 1977. A collision there between a jumbo jet trying to take off and another jet sitting on the foggy runway ended in a blazing inferno that killed 583 passengers and crew.
Building the Hardware
Arthur and his NASA colleagues made their head worn display by combining a Rockwell Collins helmet display for soldiers with their own head tracker custom built by Intersense.
The head tracker combines built in gyroscopes and a camera that detects passive paper targets pasted around a cockpit to tell exactly where a pilot has turned his or her head. The goggles can track a pilot’s head movements and overlay runways, towers, and potentially other aeroplanes over their view, which is an invaluable tool when fog rolls down across an airport.
Even as the pilot turns their head, the goggles can react in real time to ensure the virtual representations always stay in the exact same place as their real world counterparts.
Applications in Space Exploration
The ESA designed Wearable Augmented Reality (WEAR) is a wearable computer system that incorporates a head mounted display over one eye to superimpose 3D graphics and data onto its wearer’s field of view.
Controlled by voice for hands free operation, WEAR includes onboard location and object identification to show astronauts precise information about what they are looking at, as well as providing step by step instructions to guide them through difficult and lengthy procedures.
“At the moment, International Space Station (ISS) crews still use paper instruction manuals for many operational and maintenance tasks,” explained Luis Arguello of the ESA’s Modelling and Simulation Section. “Obviously, it is easier to perform a task while holding instructions in your hand. So we have developed a new type of user interface that is easier still, allowing astronauts to be guided precisely in their work without holding anything at all.”
Key hardware elements include a mobile computer connected to a headset with a head mounted display, a pair of video cameras for bar code reading and object recognition, and an Inertial Measurement Unit (IMU). The image recognition system allows WEAR to know where its user is located by checking their viewpoint against stored 3D information about the module.
