Kletzander, R. (2021):Pixel-based 2 DoF Synthesis of 360° Viewpoints with Flow-Based InterpolationVirtual Reality technology allows users to experience virtual environments by interacting with them, and navigating within them. These environments tend to be either meticulously modeled in 3D by hand, or recorded using 360° cameras. The advantage of using 360° images is that a high level of realism is achievable with relatively little effort. However, the use of 360° images generally limits users to a single viewpoint or forces them to "jump" between different viewpoints. In order to improve the viewing experience, image-based rendering, or image-based synthesis, aims to create novel viewpoints based on captured viewpoints, in the best case enabling a user to navigate freely and naturally within a scene. There are a number of different approaches to image-based synthesis, many of which use some form of feature correspondence to extract the scene geometry from the captured images. While using the scene geometry makes it possible to synthesize novel views, it can also be problematic, since inaccurate scene geometry can lead to severe artefacts, and accurate scene geometry is difficult to obtain unless dedicated depth sensors are used. In order to reduce the number and severity of these artefacts, this thesis proposes a pixel-based 2-DoF synthesis algorithm that combines basic reprojection with flow-based interpolation. Instead of estimating scene geometry, a sphere is used as a proxy for inaccurately estimated scene geometry. To mitigate the artefacts caused by the inaccurate geometry, flow-based interpolation is used to generate viewpoints with more accurate perspectives in a method called "flow-based blending". A proof-of-concept implementation of the approach is presented and tested with a select set of parameters, using different virtual and real scenes. The synthesized images are then evaluated based on mathematical error metrics, as well as on visible artefacts. The results of the evaluation show that in the majority of cases where the basic method produces significant artefacts, the synthesis using flow-based blending improves the accuracy of the results.
|