Augmented reality (AR) is a term for a live direct or indirect view of a physical real-world environment whose elements are augmented by virtual computer-generated imagery. It is related to a more general concept called mediated reality in which a view of reality is modified (possibly even diminished rather than augmented) by a computer.
In the case of Augmented Reality, the augmentation is conventionally in real-time and in semantic context with environmental elements, such as sports scores on TV during a match. With the help of advanced AR technology (e.g. adding computer vision and object recognition) the information about the surrounding real world of the user becomes interactive and digitally usable. Artificial information about the environment and the objects in it can be stored and retrieved as an information layer on top of the real world view. The term augmented reality is believed to have been coined in 1990 by Thomas Caudell, an employee of Boeing at the time.
Augmented reality research explores the application of computer-generated imagery in live-video streams as a way to expand the real-world. Advanced research includes use of head-mounted displays and virtual retinal displays for visualization purposes, and construction of controlled environments containing any number of sensors and actuators.
There are two commonly accepted definitions of Augmented Reality today. One was given by Ronald Azuma in 1997 . Azuma's definition says that Augmented Reality
Additionally Paul Milgram and Fumio Kishino defined Milgram's Reality-Virtuality Continuum in 1994 . They describe a continuum that spans from the real environment to a pure virtual environment. In between there are Augmented Reality (closer to the real environment) and Augmented Virtuality (is closer to the virtual environment).
This continuum has been extended into a two-dimensional plane of "Virtuality" and "Mediality". Taxonomy of Reality, Virtuality, Mediality. The origin R denotes unmodified reality. A continuum across the Virtuality axis V includes reality augmented with graphics (Augmented Reality), as well as graphics augmented by reality (Augmented Virtuality). However, the taxonomy also includes modification of reality or virtuality or any combination of these. The modification is denoted by moving up the mediality axis. Further up this axis, for example, we can find mediated reality, mediated virtuality, or any combination of these. Further up and to the right we have virtual worlds that are responsive to a severely modified version of reality. (at right) Mediated reality generalizes the concepts of mixed reality, etc.. It includes the virtuality reality continuum (mixing) but also, in addition to additive effects, also includes multiplicatave effects (modulation) of (sometimes deliberately) diminished reality. Moreover, it considers, more generally, that reality may be modified in various ways. The mediated reality framework describes devices that deliberately modify reality, as well as devices that accidentally modify it.
More recently, the term augmented reality has been blurred a bit due to the increased interest of the general public in AR.
Commonly known examples of AR are the yellow "first down" lines seen in television broadcasts of American football games, and the colored trail showing location and direction of the puck in TV broadcasts of ice hockey games. The real-world elements are the football field and players, and the virtual element is the yellow line, which is drawn over the image by computers in real time. Similarly, rugby fields and cricket pitches are branded by their sponsors using Augmented Reality; giant logos are inserted onto the fields when viewed on television. In some cases, the modification of reality goes beyond mere augmentation. For example, advertisements may be blocked out (partially or wholly diminished) and replaced with different advertisements. Such ad replacement is an example of Mediated reality, a concept more general AR.
Television telecasts of swimming events also often have a virtual line which indicates the position of the current world record holder at that time.
Another type of AR application uses projectors and screens to insert objects into the real environment, enhancing museum exhibitions for example. The difference to a simple TV screen for example, is that these objects are related to the environment of the screen or display, and that they often are interactive as well.
Many first-person shooter video games simulate the viewpoint of someone using AR systems. In these games the AR can be used to give visual directions to a location, mark the direction and distance of another person who is not in line of sight, give information about equipment such as remaining bullets in a gun, and display a myriad of other images based on whatever the game designers intend. This is also called the head-up display.
In some current applications like in cars or airplanes, this is usually a head-up display integrated into the windshield.
The F-35 Lightning II has no Head-up display because all targets are tracked by the aircraft's situational awareness and the sensor fusion is presented in the pilot's helmet mounted display that provides an augmented reality system that allows the pilot to look through his own aircraft as if it wasn't there.
The main hardware components for augmented reality are: display, tracking, input devices, and computer. Combination of powerful CPU, camera, accelerometers, GPS and solid state compass are often present in modern smartphones, which make them prospective platforms for augmented reality.
There are three major display techniques for Augmented Reality:
A Head Mounted Display (HMD) places images of both the physical world and registered virtual graphical objects over the user's view of the world. The HMD's are either optical see-through or video see-through in nature. An optical see-through display employs half-silver mirror technology to allow views of physical world to pass through the lens and graphical overlay information to be reflected into the user's eyes. The HMD must be tracked with a six degree of freedom sensor.This tracking allows for the computing system to register the virtual information to the physical world. The main advantage of HMD AR is the immersive experience for the user. The graphical information is slaved to the view of the user. The most common products employed are as follows: MicroVision Nomad, Sony Glasstron, and I/O Displays.
Handheld Augment Reality employs a small computing device with a display that fits in a user's hand. All handheld AR solutions to date have employed video see-through techniques to overlay the graphical information to the physical world. Initially handheld AR employed sensors such as digital compasses and GPS units for its six degree of freedom tracking sensors. This moved onto the use of fiducial marker systems such as the ARToolKit for tracking. Today vision systems such as SLAM or PTAM are being employed for tracking. Handheld display AR promises to be the first commercial success for AR technologies. The two main advantages of handheld AR is the portable nature of handheld devices and ubiquitous nature of camera phones.
Instead of the user wearing or carrying the display such as with head mounted displays or handheld devices; Spatial Augmented Reality (SAR) makes use of digital projectors to display graphical information onto physical objects. The key difference in SAR is that the display is separated from the users of the system. Because the displays are not associated with each user, SAR scales naturally up to groups of users, thus allowing for collocated collaboration between users. SAR has several advantages over traditional head mounted displays and handheld devices. The user is not required to carry equipment or wear the display over their eyes. This makes spatial AR a good candidate for collaborative work, as the users can see each other’s faces. A system can be used by multiple people at the same time without each having to wear a head mounted display. Spatial AR does not suffer from the limited display resolution of current head mounted displays and portable devices. A projector based display system can simply incorporate more projectors to expand the display area. Where portable devices have a small window into the world for drawing, a SAR system can display on any number of surfaces of an indoor setting at once. The tangible nature of SAR makes this an ideal technology to support design, as SAR supports both a graphical visualisation and passive haptic sensation for the end users. People are able to touch physical objects, and it is this process that provides the passive haptic sensation.
Modern mobile augmented reality systems use one or more of the following tracking technologies: digital cameras and/or other optical sensors, accelerometers, GPS, gyroscopes, solid state compasses, RFID, wireless sensors. Each of these technologies have different levels of accuracy and precision. Most important is the tracking of the pose and position of the user's head for the augmentation of the user's view. The user's hand(s) can tracked or a handheld input device could be tracked to provide a 6DOF interaction technique. Stationary systems can employ 6DOF track systems such as Polhemus, ViCON, A.R.T, or Ascension.
This is a current open research question. Some systems, such as the Tinmith system, employ pinch glove techniques. Another common technique is a wand with a button on it. In case of smartphone, phone itself could be used as 3D pointing device, with 3D position of the phone restored from the camera images.
Camera based systems require powerful CPU and considerable amount of RAM for processing camera images. Wearable computing systems employ a laptop in a backpack configuration. For stationary systems a traditional workstation with a powerful graphics card. Sound processing hardware could be included in augmented reality systems.
For consistent merging real-world images from camera and virtual 3D images, virtual images should be attached to real-world locations in visually realistic way. That means a real world coordinate system, independent from the camera, should be restored from camera images. That process is called Image registration and is part of Azuma's definition of Augmented Reality.
Augmented reality image registration uses different methods of computer vision, mostly related to video tracking. Many computer vision methods of augmented reality are inherited form similar visual odometry methods.
Usually those methods consist of two parts. First interest points, or fiduciary markers, or optical flow detected in the camera images. First stage can use Feature detection methods like Corner detection, Blob detection, Edge detection or thresholding and/or other image processing methods.
In the second stage, a real world coordinate system is restored from the data obtained in the first stage. Some methods assume objects with known 3D geometry(or fiduciary markers) present in the scene and make use of those data. In some of those cases all of the scene 3D structure should be precalculated beforehand. If not all of the scene is known beforehand SLAM technique could be used for mapping fiduciary markers/3D models relative positions. If no assumption about 3D geometry of the scene made structure from motion methods are used. Methods used in the second stage include projective(epipolar) geometry, bundle adjustment, rotation representation with exponential map, kalman and particle filters.
Advertising: Marketers started to use AR to promote products via interactive AR applications. For example, at the 2008 LA Auto Show, Nissan unveiled the concept vehicle Cube and presented visitors with a brochure which, when held against a webcam, showed several versions of the vehicle. In August 2009, Best Buy ran a circular with an augmented reality code that allowed users with a webcam to interact with the product in 3D.
Support with complex tasks: Complex tasks such as assembly, maintenance, and surgery can be simplified by inserting additional information into the field of view. For example, labels can be displayed on parts of a system to clarify operating instructions for a mechanic who is performing maintenance on the system. .AR can include images of hidden objects, which can be particularly effective for medical diagnostics or surgery. Examples include a virtual X-ray view based on prior tomography or on real time images from ultrasound or open NMR devices. A doctor could observe the fetus inside the mother's womb .See also Mixed reality.
Navigation devices: AR can augment the effectiveness of navigation devices for a variety of applications. For example, building navigation can be enhanced for the purpose of maintaining industrial plants. Outdoor navigation can be augmented for military operations or disaster management. Head-up displays or personal display glasses in automobiles can be used to provide navigation hints and traffic information. These types of displays can be useful for airplane pilots, too. Head-up displays are currently used in fighter jets as one of the first AR applications. These include full interactivity, including eye pointing.
Industrial Applications: AR can be used to compare the data of digital mock-ups with physical mock-ups for efficiently finding discrepancys between the two sources. It can further be employed to safeguard digital data in combination with existing real prototypes, and thus save or minimize the building of real prototypes and improve the quality of the final product.
Military and emergency services: AR can be applied to military and emergency services as wearable systems to provide information such as instructions, maps, enemy locations, and fire cells.
Prospecting: In the fields of hydrology, ecology, and geology, AR can be used to display an interactive analysis of terrain characteristics. Users could use, and collaboratively modify and analyze, interactive three-dimensional maps.
Architecture: AR can be employed to simulate planned construction projects.
Sightseeing: Models may be created to include labels or text related to the objects/places visited. With AR, users can rebuild ruins, buildings, or even landscapes as they previously existed.
Collaboration: AR can help facilitate collaboration among distributed team members via conferences with real and virtual participants. The Hand of God is a good example of a collaboration system Also see Mixed reality.
Entertainment and education: AR can be used in the fields of entertainment and education to create virtual objects in museums and exhibitions, theme park attractions (such as Cadbury World), and games (such as ARQuake and The Eye of Judgment). Also see Mixed reality.
Music: Pop group Duran Duran included interactive AR projections into their stage show during their 2000 Pop Trash concert tour. Sydney band Lost Valentinos launched the world's first interactive AR music video on 16 October 2009, where users could print out 5 markers representing a pre-recorded performance from each band member which they could interact with live and in real-time via their computer webcam and record as their own unique music video clips to share via YouTube.
It is important to note that augmented reality is a costly development in technology. Because of this, the future of AR is dependent on whether or not those costs can be reduced in some way. If AR technology becomes affordable, it could be very widespread but for now major industries are the sole buyers that have the opportunity to utilize this resource.