An overview of augmented visualization: observing the real world as desired

Abstract

Over 20 years have passed since a free-viewpoint video technology has been proposed with which a user's viewpoint can be freely set up in a reconstructed three-dimensional space of a target scene photographed by multi-view cameras. This technology allows us to capture and reproduce the real world as recorded. Once we capture the world in a digital form, we can modify it as augmented reality (i.e., placing virtual objects in the digitized real world). Unlike this concept, the augmented world allows us to see through real objects by synthesizing the backgrounds that cannot be observed in our raw perspective directly. The key idea is to generate the background image using multi-view cameras, observing the backgrounds at different positions and seamlessly overlaying the recovered image in our digitized perspective. In this paper, we review such desired view-generation techniques from the perspective of free-view point image generation and discuss challenges and open problems through a case study of our implementations.

I. INTRODUCTION

Once the real world is visually digitized via multi-view observations, the digitized world can be transferred, modified, and played on a computer from any viewpoint and time point [1,2]. Such a basic idea is called free-viewpoint image generation, and it has been over two decades since it was proposed. Using this technology, it is no longer necessary to stay in a fixed viewpoint. With a near-eye display such as head-mounted displays (HMDs), we can observe the digitized world from our egocentric viewpoint as we see the world in our daily life. Today, real-world digitization technology is within the scope of real-time processing [3], and such interactive manipulation brings us a new visualization technique: augmented visualization.

Here, we define augmented visualization as computational techniques for visualizing what cannot be seen with raw image input. While free-viewpoint image generation changes the viewpoint in a recorded virtual scene, in augmented visualization, the digitized scene is registered to the real world in accordance with our viewpoint to the presently modified vision. For example, augmented reality (AR) is considered to be an augmented visualization technique since AR overlays computer graphics (CG) (possibly CG of digitized scenes) onto the real world to change the appearance of our field of view. With augmented visualization techniques, we can modify the way we see the world.

In this paper, we give an introductory review of such visualization technologies emerging from free-viewpoint image-generation techniques that go one step beyond conventional AR. Specifically, we focus on diminished reality (DR), which has been a missing part of augmented visualization due to the lack of image and computational resources. Here, we use a case study approach to examine an easy step-up from the well-known computer vision technique (i.e., free-viewpoint image generation) to augmented visualization.

First, we briefly review the history of free-viewpoint image capture and generation (Section II). Then, we point out that abundant multi-view resources have recently become available to the public that can potentially empower one of the key technologies of augmented visualization known as DR (Section III). After giving a brief summary of [4] to describe the basics of DR (Section IV), we introduce case studies in an attempt to give readers ideas regarding how to use such resources for DR and how DR changes the way of seeing the world (Section V). Within the section, we also explain open problems in multi-view-based DR and finally summarize the discussions. Due to this, which is contrary to a recently published DR survey [4] aimed at readers in AR communities, we believe that this paper is more introductory but more convincing for readers in non-AR communities.

II. FREE-VIEWPOINT IMAGE/VIDEO

Measuring and reconstructing the three-dimensional (3D) shape of target scenes have been studied for nearly 50 years as a basic technology in the fields of image processing and measurement [5–7]. In the fields of CG and virtual reality (VR), such digitized 3D shapes have been used for taking pictures of scenes from arbitrary viewpoints where input from real-world cameras did not exist. In Virtualized Reality [1,8], Kanade et al. demonstrated that 3D shape reconstruction techniques can be applied for dynamic scenes in time-space for generating free-viewpoint videos. This technology has been studied as a new type of image presentation technology and has been used in practical situations such as sports¹.

The essence of this technology lies in how accurately one can obtain the shape and texture (colors) of the target object. However, the accuracy of the abovementioned 3D shape reconstruction-based approaches is limited and detracts from the final output image quality. On the other hand, the ray space theory-based approaches [9] can generate an arbitrary viewpoint images without acquiring the 3D shape of the scene.

Ray space theory [9], light fields [10], and lumigraphs [11] represent an image as a group of light rays in a space filled with light rays coming from every direction and has been studied since it was first proposed 20 years ago. This method is implemented by photographing the target scene with a large number of cameras (a camera array), providing a variety of viewpoints [12–14]. Recently proposed methods using such image data structures can computationally change camera parameters, such as focus and aperture, after the shooting whereas previously we had to change the parameters before the shooting when using conventional cameras [15,16]. Also, methods for photographing pictures in higher resolutions, in terms of space and time beyond the limitations determined by sampling theory, have been proposed [17,18]. Thus, future developments in computational photography technology are expected.

III. FREE-VIEWPOINT TO CONTROLLABLE REALITY

The techniques in the previous section are used to collect image data photographed and acquired by sensors, such as cameras, or to generate desirable images for observers after the data acquisition. For example, when free-viewpoint video technology is used in broadcasting what free usually means is that the observer is free from the observation position that the photographer or the director of the broadcast had been predetermined [21–23]. Therefore, free-viewpoint videos allow the audience actually watching the video to move their viewing positions to where they desire (see Figs 1(a)–1(c)).

Fig. 1. Differences in conventional photos or videos, free-viewpoint image generation, and augmented visualization. After recording the scene (a), the observer's viewpoint is fixed in the conventional photos or videos (b). Free-viewpoint image generation enables the observer to move around in the recorded virtual space through a physically fixed display (c). In augmented visualization (d), with wearable displays such as near-eye displays [19,20], the observers can move around in the real space to which the recorded virtual scene (objects with dotted lines) is registered as proxies for visual augmentation. In other words, we could achieve free scene modifications using both AR and DR technologies via the proxies (e.g., the star-shaped object is added and the box is removed, respectively, in this example).

In other words, this is a technology that allows us to change our viewpoint in the digitized 3D world, regardless of the intention of the direct acquisition of the image data. The technique for such desirable images, regardless of the intention or situations at the time of actual image capture, becomes more meaningful as the amount of acquired image data increases. When the camera was invented, “taking a picture” required considering every property every time (camera position, posture, subject, shutter, etc.) since the cost required for one photograph was large. Therefore, even a single photo conveyed details inexpressible by words to many people.

In the modern worlds, plenty of surveillance cameras are installed in societies, and images are collected at every moment (e.g., “EarthCam”² and “CAM²”³). Similarly, many images are taken by individuals with their personal cameras (e.g., Instagram⁴). However, most of the surveillance and personal camera images are not seen by humans. That is, a lot of images are stored that are not seen by anyone. Therefore, there is a demand for techniques to re-create and present images from these huge image datasets in desired forms [24–26]. As an example of such forms, the authors have been conducting research on image generation of DR [4], which is one of the missing components of augmented visualization. In contrast to AR, in DR such image resources are synthesized from the user's perspective in accordance with the user's position to virtually reveal the hidden parts of the scene (Fig 1(d)). As we explained in Section I, with a combination of AR and DR, we can freely modify the captured environment. In other words, combining AR and DR completes augmented visualization.

IV. DIMINISHED REALITY: PRINCIPLES

A) Comparing DRs

DR is different from AR [27–29], which superimposes virtual objects on the real world to enhance reality. AR overlays virtual objects to add positive information to the real world. DR overlays virtual objects as well, but the objects are negative information to offset the reality. Therefore, we could say that DR is a visualization method based on diminishing.

Figure 2 shows schematic figures describing differences in the real world, VR, AR, and DR in terms of visible light rays in each reality display. In (a), the observer sees real objects in the environment. In VR (b), HMD occludes light rays of the real environment and presents ones from the virtual world. In AR (c), real light rays are visible through the HMD and, at the same time, virtual rays are presented. Thus, the virtual object (black star) looks like it is existing in the real environment. In DR (d), real rays are selectively occluded by the HMD. Rays initially occluded by objects are recovered and therefore visible through the HMD. To achieve this, the digitized scene must be registered to the real one and tracked according to the observer's head motion. Here, DR requires free-viewpoint image-generation technology.

Fig. 2. Observable light rays in (a) the real space, (b) VR, (c) AR, and (d) DR. Light rays are selectively presented to the eyes via an HMD. In DR, occluded light rays (the blue rays) from the user's viewpoint must be observed and reproduced based on the other image resources. Here, we need free-viewpoint image-generation techniques.

B) Background resources

Figure 3 shows typical settings in DR. DR methods assume that the background resources are obtained by (1) directly observing the background from different viewpoints (e.g., surveillance cameras and other users' cameras), (2) copying and pasting pixels of patches from the surrounding regions or previous frames, or (3) fetching images from databases such as Google Street View. (4) Of course, we can combine these approaches. Based on the available background resources, we can categorize DR methods [4]. Note that DR relies on free-viewpoint image-generation techniques when it employs these approaches (except the second one).

Fig. 3. Background resources in DR. The goal of DR is to estimate background from (a) different perspectives, (b) surrounding pixels, (c) database resources, and these combination.

C) Functions

DR technology is defined as a set of methodologies (diminishing, replacing, in-painting, or making something see-through) for altering real objects in a perceived environment in real time to “diminish” reality [4]. Each function can be described as follows: to diminish objects, the objects of interest are degraded in colors or textures to get less attention; to make objects see-through, the backgrounds of the objects are digitized beforehand or in real time in a similar manner to free-viewpoint image generation and overlaid onto the observer's view in accordance with the observer's head motion; to replace objects, alternative virtual objects are overlaid onto the real objects to hide them; to in-paint objects, plausible background images are generated on the fly from all pixels except for those in the region of interest (ROI).

Figures 4 and 5 show example results of see-through and in-paint, respectively. Note that in-paint does not require resources for different perspectives (review the second approach in Section IV B), and therefore, in-paint [30–32] does not visualize the “real” background. The resultant visualization is computationally generated, and there is no guarantee that the actual hidden areas will be visualized. On the other hand, see-through uses the free-viewpoint image-generation technology to visualize the hidden areas as they are [33–37].

Fig. 4. See-through approach. Multi-view image resources (a), user view (b), and DR view (c).

Fig. 5. In-paint approach. Input masked image (a), position map (b), and DR view (c). A position map indicates which pixels in the surrounding regions should be mapped to the region of interest. In this example, each color in the region of interest corresponds to pixels of the same color in the surrounding pixels.

D) Display devices

Display devices are one of the important factors in DR. Most DR researches implicitly assume video see-through systems since optical see-through devices cannot perfectly occlude incoming lights from the real environments due to their see-through nature [38–40].

Egocentric displays, such as HMDs, are ideal in terms of portable and hands-free experiences. Tablet systems can also be used as egocentric displays, although user-perspective distortion should be corrected to compensate for differences in viewpoint between the recording camera and the eyes [41–44]. Since the tablet surface looks transparent following distortion correction, user-perspective distortion correction is considered to be a DR application.

V. CASE STUDY OF FREE-VIEWPOINT IMAGE-GENERATION-BASED DR

A) Overview

In this section, we introduce four DR implementations as case studies where free-viewpoint image-generation techniques are necessary to address challenges in DR. Here, we show how augmented visualization changes the way of seeing the world and how free-viewpoint image-generation techniques contribute to augmented visualization. Keeping in mind the background of this paper, here we show see-through approaches only. We additionally discuss practical issues related to each case to highlight open problems in DR. Table 1 provides a summary of the case studies.

Table 1. Summary of the case studies. These four examples are presented in Section V to discuss general issues in DR

B) Filmmaking support [45]

1) Background and challenges

Special effects (SFX) or visual effects (VFX) help to make films popular, but the filmmaking process is complicated. Therefore, pre-visualized movies created in the early stage of the filmmaking, referred to as PreVis, have played an important role over the last several decades. PreVis movies are usually shot with low-cost cameras or generated using simple CG to share a creator's vision for directing staff and to estimate camera movements, necessary personnel, and other potential expenditures.

However, newly placed vending machines or modern signboards in the environment can ruin the image, especially when filmmakers create a historical movie. This is where we apply DR to remove such objects in PreVis. In this filmmaking scenario, we have numerous photos as resources recorded in advance in the location-hunting stage (e.g., several weeks ago). The scenes in such photos should be similar to the views to be filmed in PreVis, while the only problem is that the pictures were taken at different time points. In this example, we had to augment images in multiple monitors to share the visualizations among crews members. One had to be a portable monitor attached to a cinematographic camera for a cameraman. Fig. 6(a) illustrates this issue.

Fig. 6. Filmmaking support. (a) The ideal scene is recorded in advance to be used in the PreVis. The illumination changes and, consequently, the replaced scene appears inconsistent when seen through the object of interest (the square with dotted lines). (b) The construction signboard is replaced with a pre-recorded scene captured in different lighting. The red arrow indicates photometric borders of the ROI [45].

2) Example solution

Since the appearance of the photos is quite similar to the views in PreVis, we do not generate a new image from the photo resources, although we need to align the photos to the views in PreVis via homography warping. We can simply select and warp one of these images onto the video stream from a cinematography camera to hide the newly placed visually annoying object.

However, when the background image is synthesized to overlay the object, photometric inconsistency between the synthesized and the surrounding regions occurs. Since the illumination conditions are different from each other, the simple overlay generates conspicuous borders around the ROI, as shown in Fig. 6(b). In augmented visualization, keeping visual coherence between the real and synthesized images is one of the most important issues and is known as “photometric consistency”. These borders need to be reduced using a color compensation technique.

DR methods do not estimate illuminations directly on the site but instead use color tone correction processing on the image plane to achieve real-time processing [32,49]. Even if the light source always moves around within a 3D scene, a simple color tone correction often results in sufficient results with regard to the appearance unless the processing is performed in real time [50].

3) Additional discussion: registration

Every DR methods begin with registering the digitized scene to the real scene (recall Section 3(d)). This filmmaking support scenario was the simplest case with regard to the registration, as explained above. When an object to be removed is small or far from the observer, 2D alignment may be sufficient even for 3D background scenes [49]. When the object to be removed is large or close to the observer (as in the case here), the background region occluded by the object that must be removed increases, and 3D registration is required.

Visual simultaneous localization and mapping (vSLAM) [51] is sufficient for the 3D alignment, but an extension for registration with the background's and vSLAM's coordinates is required. There is also an efficient 3D registration method that utilizes the previously [50] or currently obtained [46] textured 3D mesh of the background. Since the result of the synthesis on the screen space is the final output for DR, matching the generated background and the current image should be effective.

C) Blind spot visualization [46]

1) Background and challenges

Blind spot visualization techniques are known as AR X-ray vision [33,52] or see-through vision [35]. These visualization techniques enable observers to see through walls like Superman, as their names suggest. Using this technology, for example, we can check the cars and pedestrians approaching from behind walls and can check whether the shops behind buildings are open or not. Although we can see behind walls using curved mirrors, see-through technology does not require us to mentally transform the view in 3D like we usually do. To provide such views, we need to show DR results in portable displays, such as a smartphone or a head-up display (HUD) in a car.

We can achieve the see-through ability by overlaying real-time video feeds from cameras placed behind walls onto the observer's video. However, in this setting, the field of view of the background observation cameras is potentially limited because they are basically placed ahead of the user's camera, as illustrated in Fig. 7(a). Conventional approaches use multiple surveillance cameras to cover the whole background, although this setup makes things difficult with regard to recording time synchronizations, color compensations, and calibrations between the installed cameras.

Fig. 7. Blind spot visualization. (a) Blind spot visualization potentially requires multiple cameras to cover a wide area. (b) A single wide field of view RGB-D camera makes the setting simpler and can mitigate a number of issues in (a). (c) Real-time blind spot visualization with the RGB-D camera [46].

2) Example solution

To overcome this problem, as a practical solution, we proposed to build a wide field of view RGB-D camera combining a range scanner and an affordable and widely available fish-eye camera [46] (Figs 7(b) and 7(c)). The single wide field of view RGB-D camera straightforwardly mitigates the above-mentioned issues in blind spot visualization. Given such RGB-D images, we can create a wide range of textured meshes in real-time. This is because the reconstructed view is wide enough to have extra regions for the user's camera to match and register the 3D textured mesh to the user's camera image (Fig 7(c)).

3) Additional discussion: depth perception

Given the user's camera pose, free-viewpoint image-generation techniques provide synthetic views based on the resources. However, simply overlaying the registered free-viewpoint image is not acceptable since all information about the front objects will be lost. A straightforward method to solve this problem is to alpha blend an original input image and a diminished reality image. However, the resulting image will have two types of information (i.e., background and foreground) at once. Therefore, effective representation methods of this information have been discussed in the AR X-ray vision area.

Considering DR as an information filtering method would also involve ethical problems. Therefore, it is a difficult problem to decide what to hide or show and whether to automatically control or trust the user. Although erasing a vehicle in front will give us a wide field of view, it is easy to imagine that we will collide with the invisible vehicle. The perceived appearance varies greatly depending on when the forward vehicle is displayed semi-transparently, when the focus is on the front vehicle, or when the focus is on the far road. Therefore, besides DR displays, taking into account human sensing technologies such as accommodation estimation will become important.

D) Work area visualization [47]

1) Background and challenges

Handcraft operators might see the work area when their move their heads, but they may have difficulty when the holding tool is large or when a third person watching the first person's view video cannot see the work area since he or she cannot move the viewing position. Using DR technology, the operator can change the transparency of hands and tools in the operator's view to check the working process. Here, all of the processes from video capture to visualization should run in interactive rates since the scene is completely dynamic. Such ergonomic views can be provided through HMDs.

2) Example solution

Figure 4 shows an example of visualizing a work area occluded by an operator's hand and a holding tool in the perspective [47,53]. In this setting, we had no space to place cameras below the hand or the tool, and therefore, we surrounded the workspace to capture the whole area. This setting reproduces a virtual camera with a huge aperture (i.e., we record the light field in real time). During the rendering of the synthetic camera, we give less weight to light rays passing through the hand and the tools that are not to be reproduced in the rendering (Fig 8).

Fig. 8. Work area visualization. The rays are reproduced by the rays with dotted lines that ignore the hand [53].

The rendering approach [53] named detour light field rendering (LFR) is computationally efficient since no precise ROI segmentation and tracking are required, which are usually computationally expensive. Instead, a point representing an object to be removed is detected or manually placed. This representative point determines an effective range and excludes light rays passing around the point from the rendering, as illustrated in Fig. 8. In other words, this rendering process mitigates segmentation processing, which is usually computationally expensive.

E) Additional discussion: effects of DR

Pseudo-haptics [54] is a famous perceptual concept suggesting that our haptic sensations are easily affected by a vision in VR. It is known that this perceptual illusion is also present in AR scenarios [55,56]. In this context, what happens in DR scenarios when a user's hand or held object is removed from the user's perspective has not been well discussed. We obtained some user study results demonstrating that virtually shortened sticks can feel heavier than they actually are [57].

F) De-fencing [48]

1) Background and challenges

De-fencing refers to techniques to diminish the appearance of a fence in an image to create a fence-free view [48]. Such a technique is useful, for example, when a photographer takes a photo of a tourist landmark but the scene is occluded by fences for security reasons. The resultant de-fenced videos can be provided on a smartphone display through streaming or in a recorded video format. The challenge here is to segment the fences and ignore them in the free-viewpoint image generation.

2) Example solution

This task can be achieved following a similar approach to the light field rendering-based DR in Section V-D(Fig 9). Here, we set points at the fenceless weights to replace the fence pixels with the pixels in the other images recorded while sweeping a camera. However, in this case, we cannot represent the fence with several points since the fence pixels appear in the whole image. Instead, we segment a 3D scene reconstruction into fence and non-fence points since fence points always appear closer to the camera.

Fig. 9. De-fencing. (a) The rays with solid lines are reproduced by the rays with dotted lines that ignore the fence points. (b) Consequently, the scene without the fence is restored [48].

3) Additional discussion: object selection

Considering practical scenarios, the object detection task in DR seems rather challenging compared to those in de-fencing and work area visualization tasks, as the object to be removed is usually unknown until the observer faces it or it is determined based on the user's preference. Therefore, objects to be removed are selected through user interaction, such as clicking and dragging.

When selecting an object to be removed through user interaction [30,32,50], the region must be properly enclosed, and even after that, it must keep tracking along with the movement of the viewpoint. Many methods cover objects with a 3D bounding object [32,50] or employ image-based tracking methods [30,49]. One state-of-the-art method enables users to select objects through a category consisting of a combination of a convolutional neural network and vSLAM [58]. For example, the users can remove “trash” or “furniture” from the environment just by selecting the categories.

VI. CONCLUSIONS

In this paper, the authors discussed the features of arbitrary viewpoint image generation and extended idea to augmented visualization. All examples shown in Section V are based on or closely related to principles of the two approaches described in Section II (i.e., 3D shape reconstruction and ray space theory).

The essence of these approaches are identical in that their purpose is to describe the 3D space regardless of differences in data representation. In DR, the described 3D space is arbitrarily reproduced in accordance with the real space coordinates to see the scene in the desired way. As described in the case study in Section V, DR has special issues to be solved in terms of image processing, such as viewpoint changes with limited image and computational resources under variable lighting conditions. While some DR methods achieved high-quality object removal results, researchers do not know how the results are perceived. The authors feel that these perceptual issues need further discussion in the near future.

The authors would like to introduce the first international DR survey paper for readers who discovered DR technology through this paper [4]. The authors are the founding members of the “Technical Committee on Plenoptic Time-Space Technology (PoTS)⁵ ”, founded in Japan in 2015. PoTS started to investigate a way of presentation via spatio-time representation based on the idea of extending ray space theory to the time axis. Through such research activities, the authors hope to further advance the technology.