Augmented reality

3D tracking

AR systems estimate device pose and scene geometry so virtual graphics stay aligned with the real world. Common approaches include visual–inertial odometry and SLAM for markerless tracking, and fiducial markers when known patterns are available; image registration and depth cues (e.g., occlusion, shadows) maintain realism.

Software and standards

AR runtimes provide sensing, tracking, and rendering pipelines; mobile platforms expose SDKs with camera access and spatial tracking. Interchange/geospatial formats such as ARML standardize anchors and content.

Interaction and input

Input commonly combines head/gaze with touch, controllers, voice, or hand tracking; audio and haptics can reduce visual load. Human-factors studies report performance benefits but also workload and safety trade-offs depending on task and context.

Design considerations

Key usability factors include stable registration, legible contrast under varied lighting, and low motion-to-photon latency. Visual design often uses depth cues (occlusion, shadows) to support spatial judgment; safety-critical uses emphasize glanceable prompts and minimal interaction.

Comparison with mixed reality/virtual reality

Augmented reality (AR) is largely synonymous with mixed reality (MR). There is also overlap in terminology with extended reality and computer-mediated reality. However, In the 2020s, the differences between AR and MR began to be emphasized.

In augmented reality, users are not only able to view digital content within their real environment but can also interact with it as if it were a tangible part of the physical world. This is made possible through devices such as Meta Quest 3S and Apple Vision Pro, which utilize multiple cameras and sensors to enable real-time interaction between virtual and physical elements. Mixed reality that incorporates haptics has sometimes been referred to as visuo-haptic mixed reality.

In virtual reality (VR), the users' perception is completely computer-generated, whereas with augmented reality (AR), it is partially generated and partially from the real world. For example, in architecture, VR can be used to create a walk-through simulation of the inside of a new building; and AR can be used to show a building's structures and systems super-imposed on a real-life view. Another example is through the use of utility applications. Some AR applications, such as Augment, enable users to apply digital objects into real environments, allowing businesses to use augmented reality devices as a way to preview their products in the real world. Similarly, it can also be used to demo what products may look like in an environment for customers, as demonstrated by companies such as Mountain Equipment Co-op or Lowe's who use augmented reality to allow customers to preview what their products might look like at home.

Augmented reality (AR) differs from virtual reality (VR) in the sense that in AR, the surrounding environment is real and AR is just adding virtual objects to the real environment. On the other hand, in VR, the surrounding environment is completely virtual and computer generated. A demonstration of how AR layers objects onto the real world can be seen with augmented reality games. WallaMe is an augmented reality game application that allows users to hide messages in real environments, utilizing geolocation technology in order to enable users to hide messages wherever they may wish in the world.

The use of the terms "mixed reality" and "interreality" is clearly defined in the context of physics and may be slightly different in other fields, however, it is generally seen as, "bridging the physical and virtual world".

Recent improvements in AR and VR headsets have made the display quality, field of view, and motion tracking more accurate, which makes augmented experiences more immersive. Improvements in sensor calibration, lightweight optics, and wireless connectivity have also made it easier for users to move around and be comfortable.

History

Precursors to augmented reality

• 1901: Author L. Frank Baum, in his science-fiction novel The Master Key, first mentions the idea of an electronic display/spectacles that overlays data onto real life (in this case 'people'). It is named a 'character marker'.
• Heads-up displays (HUDs), a precursor technology to augmented reality, were first developed for pilots in the 1950s, projecting simple flight data into their line of sight, thereby enabling them to keep their "heads up" and not look down at the instruments. It is a transparent display.

Earliest developments

• 1968: Ivan Sutherland creates the first optical-see through head-mounted display that has graphics rendered by a computer.
• 1975: Myron Krueger creates Videoplace to allow users to interact with virtual objects.
• 1980: The research by Gavan Lintern of the University of Illinois is the first published work to show the value of a heads up display for teaching real-world flight skills.
• 1980: Steve Mann creates the first wearable computer, a computer vision system with text and graphical overlays on a photographically mediated scene.
• 1986: Within IBM, Ron Feigenblatt describes the most widely experienced form of AR today (viz. "magic window," e.g. smartphone-based Pokémon Go), use of a small, "smart" flat panel display positioned and oriented by hand.
• 1987: Douglas George and Robert Morris create a working prototype of an astronomical telescope-based "heads-up display" system (a precursor concept to augmented reality) which superimposed in the telescope eyepiece, over the actual sky images, multi-intensity star, and celestial body images, and other relevant information.
• 1988: David Drascic, Paul Milgram, and Julius Grodski demonstrate their working prototype of an interactive stereoscopic video system with superimposed stereoscopic graphics, which allows the user to select points in the video view for path planning, target acquisition, or measurement with a "virtual tape-measure".
• 1990: The term augmented reality is attributed to Thomas P. Caudell, a former Boeing researcher.
• 1992: Louis Rosenberg developed one of the first functioning AR systems, called Virtual Fixtures, at the United States Air Force Research Laboratory—Armstrong, that demonstrated benefit to human perception.
• 1992: Steven Feiner, Blair MacIntyre and Doree Seligmann present an early paper on an AR system prototype, KARMA, at the Graphics Interface conference.
• 1993: George Fitzmaurice created the first handheld spatially aware display for providing virtual and augmented experiences.
• 1993: Mike Abernathy, et al., report the first use of augmented reality in identifying space debris using Rockwell WorldView by overlaying satellite geographic trajectories on live telescope video.
• 1993: A widely cited version of the paper above is published in Communications of the ACM – Special issue on computer augmented environments, edited by Pierre Wellner, Wendy Mackay, and Rich Gold.
• 1994: Andrei State et al. present a prototype medical AR application, enabling physicians to observe a fetus within a pregnant patient.
• 1994: Paul Milgram et al define Mixed Reality, and emphasize that Augmented Reality is only one possibility in the Reality-Virtuality Continuum.
• 1995: S. Ravela et al. at University of Massachusetts introduce a vision-based system using monocular cameras to track objects (engine blocks) across views for augmented reality.
• 2004: An outdoor helmet-mounted AR system was demonstrated by Trimble Navigation and the Human Interface Technology Laboratory (HIT lab).

Smartphone AR and modern headsets

• 2009: ARToolkit was ported to Adobe Flash (FLARToolkit) by Saqoosha, bringing augmented reality to the web browser.
• 2015: Microsoft announced the HoloLens augmented reality headset, which uses various sensors and a processing unit to display virtual imagery over the real world.
• 2015: Apple quietly acquired Metaio, a Munich-based augmented reality pioneer founded in 2003, integrating its computer-vision patents and engineering team into what would later become ARKit.
• 2016: Niantic released Pokémon Go for iOS and Android in July 2016. The game quickly became one of the most popular smartphone applications and in turn spikes the popularity of augmented reality games.
• 2018: Magic Leap launched the Magic Leap One augmented reality headset. Leap Motion announced the Project North Star augmented reality headset, and later released it under an open source license.
• 2019: Microsoft announced HoloLens 2 with significant improvements in terms of field of view and ergonomics.
• 2022: Magic Leap launched the Magic Leap 2 headset.
• 2023: Meta Quest 3, a mixed reality VR headset was developed by Reality Labs, a division of Meta Platforms. In the same year, Apple Vision Pro was released.
• 2024: Meta Platforms revealed the Orion AR glasses prototype.

Uses

Augmented reality has been explored for many uses, including education and business. Some of the earliest cited examples include augmented reality used to support surgery by providing virtual overlays to guide medical practitioners, to AR content for astronomy and welding. Example application areas described below include archaeology, architecture, commerce and education.

Education and training

AR for education and training can overlay 3D models and step-by-step guidance in real settings (e.g., anatomy, maintenance); systematic reviews report learning benefits alongside design and implementation caveats that vary by context and task.

Navigation and maps

Augmented reality navigation overlays route guidance or hazard cues onto the real scene, typically via smartphone "live view" or in-vehicle heads-up displays. Research finds AR can improve wayfinding and driver situation awareness, but human-factors trade-offs (distraction, cognitive load, occlusion) matter for safety-critical use.

See also: Head-up display, Automotive navigation system, Wayfinding

Commerce

In 2021, iBite was one of the first iOS applications to integrate Apple's ARKit & RealityKit Swift frameworks for interactive augmented reality digital ordering. iBite allows users to view 3D models of their food before ordering, and allow merchants to upload their own USDZ files which they can generate using iBite's patented photogrammetry software.

In 2018, Apple announced USDZ, a file format based on Universal Scene Description from Pixar, which allows 3D objects to be viewed in AR on iPhones and iPads with iOS 12. Apple has created an AR QuickLook Gallery that allows people to experience augmented reality through their own Apple device.

In 2018, Shopify, the Canadian e-commerce company, announced AR Quick Look integration. Their merchants will be able to upload 3D models of their products and their users will be able to tap on the models inside the Safari browser on their iOS devices to view them in their real-world environments.

AR technology is used by furniture retailers such as IKEA, Houzz, and Wayfair. These retailers offer apps that allow consumers to view their products in their home prior to purchasing anything.

In 2017, Ikea announced the Ikea Place app. It contains a catalogue of over 2,000 products—nearly the company's full collection of sofas, armchairs, coffee tables, and storage units which one can place anywhere in a room with their phone. The app made it possible to have 3D and true-to-scale models of furniture in the customer's living space. IKEA realized that their customers are not shopping in stores as often or making direct purchases anymore. Shopify's acquisition of Primer, an AR app aims to push small and medium-sized sellers towards interactive AR shopping with easy to use AR integration and user experience for both merchants and consumers. AR helps the retail industry reduce operating costs. Merchants upload product information to the AR system, and consumers can use mobile terminals to search and generate 3D maps.

Surgery

One of the first applications of augmented reality was in healthcare, particularly to support the planning, practice, and training of surgical procedures. As far back as 1992, enhancing human performance during surgery was a formally stated objective when building the first augmented reality systems at U.S. Air Force laboratories. AR provides surgeons with patient monitoring data in the style of a fighter pilot's heads-up display, and allows patient imaging records, including functional videos, to be accessed and overlaid. Examples include a virtual X-ray view based on prior tomography or on real-time images from ultrasound and confocal microscopy probes, visualizing the position of a tumor in the video of an endoscope, or radiation exposure risks from X-ray imaging devices. AR can enhance viewing a fetus inside a mother's womb. Siemens, Karl Storz and IRCAD have developed a system for laparoscopic liver surgery that uses AR to view sub-surface tumors and vessels.

Guidance overlays and image fusion support planning and intraoperative visualization across several specialties; reviews note accuracy/registration constraints and workflow integration issues.

The HoloLens is capable of displaying images for image-guided surgery. As augmented reality advances, it finds increasing applications in healthcare. Augmented reality and similar computer based-utilities are being used to train medical professionals. In healthcare, AR can be used to provide guidance during diagnostic and therapeutic interventions e.g. during surgery. Magee et al., for instance, describe the use of augmented reality for medical training in simulating ultrasound-guided needle placement. Recently, augmented reality began seeing adoption in neurosurgery, a field that requires heavy amounts of imaging before procedures.

Smartglasses can be incorporated into the operating room to aide in surgical procedures; possibly displaying patient data conveniently while overlaying precise visual guides for the surgeon. Augmented reality headsets like the Microsoft HoloLens have been theorized to allow for efficient sharing of information between doctors, in addition to providing a platform for enhanced training. This can, in some situations (i.e. patient infected with contagious disease), improve doctor safety and reduce PPE use. While mixed reality has lots of potential for enhancing healthcare, it does have some drawbacks too. The technology may never fully integrate into scenarios when a patient is present, as there are ethical concerns surrounding the doctor not being able to see the patient. Mixed reality is also useful for healthcare education. For example, according to a 2022 report from the World Economic Forum, 85% of first-year medical students at Case Western Reserve University reported that mixed reality for teaching anatomy was "equivalent" or "better" than the in-person class.

Flight training

Building on decades of perceptual-motor research in experimental psychology, researchers at the Aviation Research Laboratory of the University of Illinois at Urbana–Champaign used augmented reality in the form of a flight path in the sky to teach flight students how to land an airplane using a flight simulator. An adaptive augmented schedule in which students were shown the augmentation only when they departed from the flight path proved to be a more effective training intervention than a constant schedule. Flight students taught to land in the simulator with the adaptive augmentation learned to land a light aircraft more quickly than students with the same amount of landing training in the simulator but with constant augmentation or without any augmentation.

Military

The first augmented reality system that integrated haptic 3D input was the Virtual Fixtures platform, which was developed in 1992 by Louis Rosenberg at the Armstrong Laboratories of the United States Air Force. It enabled human users to control robots in real-world environments using a haptic controller. Published studies showed that by introducing virtual objects into the real world, significant performance increases could be achieved by human operators.

An interesting early application of AR occurred when Rockwell International created video map overlays of satellite and orbital debris tracks to aid in space observations at Air Force Maui Optical System. In their 1993 paper "Debris Correlation Using the Rockwell WorldView System" the authors describe the use of map overlays applied to video from space surveillance telescopes. The map overlays indicated the trajectories of various objects in geographic coordinates. This allowed telescope operators to identify satellites, and also to identify and catalog potentially dangerous space debris.

Starting in 2003 the US Army integrated the SmartCam3D augmented reality system into the Shadow Unmanned Aerial System to aid sensor operators using telescopic cameras to locate people or points of interest. The system combined fixed geographic information including street names, points of interest, airports, and railroads with live video from the camera system. The system offered a "picture in picture" mode that allows it to show a synthetic view of the area surrounding the camera's field of view. This helps solve a problem in which the field of view is so narrow that it excludes important context, as if "looking through a soda straw". The system displays real-time friend/foe/neutral location markers blended with live video, providing the operator with improved situational awareness.

Combat reality can be simulated and represented using complex, layered data and visual aides, most of which are head-mounted displays (HMD), which encompass any display technology that can be worn on the user's head. Military training solutions are often built on commercial off-the-shelf (COTS) technologies, such as Improbable's synthetic environment platform, Virtual Battlespace 3 and VirTra, with the latter two platforms used by the United States Army. As of 2018, VirTra is being used by both civilian and military law enforcement to train personnel in a variety of scenarios, including active shooter, domestic violence, and military traffic stops.

In 2017, the U.S. Army was developing the Synthetic Training Environment (STE), a collection of technologies for training purposes that was expected to include mixed reality. As of 2018, STE was still in development without a projected completion date. Some recorded goals of STE included enhancing realism and increasing simulation training capabilities and STE availability to other systems.

It was claimed that mixed-reality environments like STE could reduce training costs, such as reducing the amount of ammunition expended during training. In 2018, it was reported that STE would include representation of any part of the world's terrain for training purposes. STE would offer a variety of training opportunities for squad brigade and combat teams, including Stryker, armory, and infantry teams.

Researchers at USAF Research Lab (Calhoun, Draper et al.) found an approximately two-fold increase in the speed at which UAV sensor operators found points of interest using this technology. This ability to maintain geographic awareness quantitatively enhances mission efficiency. The system is in use on the US Army RQ-7 Shadow and the MQ-1C Gray Eagle Unmanned Aerial Systems.

In combat, AR can serve as a networked communication system that renders useful battlefield data onto a soldier's goggles in real time. From the soldier's viewpoint, people and various objects can be marked with special indicators to warn of potential dangers. Virtual maps and 360° view camera imaging can also be rendered to aid a soldier's navigation and battlefield perspective, and this can be transmitted to military leaders at a remote command center. The combination of 360° view cameras visualization and AR can be used on board combat vehicles and tanks as circular review system.

AR can be an effective tool for virtually mapping out the 3D topologies of munition storages in the terrain, with the choice of the munitions combination in stacks and distances between them with a visualization of risk areas. The scope of AR applications also includes visualization of data from embedded munitions monitoring sensors.

Navigation

The NASA X-38 was flown using a hybrid synthetic vision system that overlaid map data on video to provide enhanced navigation for the spacecraft during flight tests from 1998 to 2002. It used the LandForm software which was useful for times of limited visibility, including an instance when the video camera window frosted over leaving astronauts to rely on the map overlays. The LandForm software was also test flown at the Army Yuma Proving Ground in 1999. In the photo at right one can see the map markers indicating runways, air traffic control tower, taxiways, and hangars overlaid on the video.

Industrial environments

In industrial environments, augmented reality is proving to have a substantial impact with use cases emerging across all aspect of the product lifecycle, starting from product design and new product introduction (NPI) to manufacturing to service and maintenance, to material handling and distribution. For example, labels were displayed on parts of a system to clarify operating instructions for a mechanic performing maintenance on a system. Assembly lines benefited from the usage of AR. In addition to Boeing, BMW and Volkswagen were known for incorporating this technology into assembly lines for monitoring process improvements. Big machines are difficult to maintain because of their multiple layers or structures. AR permits people to look through the machine as if with an x-ray, pointing them to the problem right away.

Functional mockup

Augmented reality can be used to build mockups that combine physical and digital elements. With the use of simultaneous localization and mapping (SLAM), mockups can interact with the physical world to gain control of more realistic sensory experiences like object permanence, which would normally be infeasible or extremely difficult to track and analyze without the use of both digital and physical aides.

Translation

AR applications such as Word Lens can interpret the foreign text on signs and menus and, in a user's augmented view, re-display the text in the user's language. Spoken words of a foreign language can be translated and displayed in a user's view as printed subtitles.

Human-in-the-loop operation of robots

Recent advances in mixed-reality technologies have renewed interest in alternative modes of communication for human-robot interaction. Human operators wearing augmented reality headsets such as HoloLens can interact with (control and monitor) e.g. robots and lifting machines on site in a digital factory setup. This use case typically requires real-time data communication between a mixed reality interface with the machine / process / system, which could be enabled by incorporating digital twin technology.

Real life ad-blocking

More than one in three surveyed advanced Internet users would like to edit out disturbing elements around them, such as garbage or graffiti. They would like to even modify their surroundings by erasing street signs, billboard ads, and uninteresting shopping windows. Consumers want to use augmented reality glasses to change their surroundings into something that reflects their own personal opinions. Around two in five want to change the way their surroundings look and even how people appear to them.

Apps

Snapchat users have access to augmented reality features. In September 2017, Snapchat announced a feature called "Sky Filters" that will be available on its app. This new feature makes use of augmented reality to alter the look of a picture taken of the sky, much like how users can apply the app's filters to other pictures. Users can choose from sky filters such as starry night, stormy clouds, beautiful sunsets, and rainbow.

Google launched an augmented reality feature for Google Maps on Pixel phones that identifies users' location and places signs and arrows on the device screen to show a user navigation directions.

Concerns

Accidents

In a paper titled "Death by Pokémon GO", researchers at Purdue University's Krannert School of Management claim the game caused "a disproportionate increase in vehicular crashes and associated vehicular damage, personal injuries, and fatalities in the vicinity of locations, called PokéStops, where users can play the game while driving." Using data from one municipality, the paper extrapolates what that might mean nationwide and concluded "the increase in crashes attributable to the introduction of Pokémon GO is 145,632 with an associated increase in the number of injuries of 29,370 and an associated increase in the number of fatalities of 256 over the period of 6 July 2016, through 30 November 2016." The authors extrapolated the cost of those crashes and fatalities at between $2bn and $7.3 billion for the same period.

Privacy concerns

Augmented reality devices that use cameras for 3D tracking or video passthrough depend on the ability of the device to record and analyze the environment in real time. Because of this, there are potential legal concerns over privacy.

According to recent studies, users are especially concerned that augmented reality smart glasses might compromise the privacy of others, potentially causing peers to become uncomfortable or less open during interactions.

Notable researchers

• Ronald Azuma is a scientist and author of works on AR.
• Steve Mann formulated an earlier concept of mediated reality in the 1970s and 1980s, using cameras, processors, and display systems to modify visual reality to help people see better (dynamic range management), building computerized welding helmets, as well as "augmediated reality" vision systems for use in everyday life. He is also an adviser to Meta.
• Dieter Schmalstieg and Daniel Wagner developed a marker tracking systems for mobile phones and PDAs in 2009.
• Ivan Sutherland invented the first augmented reality system, often called The Sword of Damocles, at Harvard University.