Navigating Through Sparse Views

Proceedings of VRST (Symposium on Virtual Reality Software and Technology) '99

Abstract

This paper presents an image-based walkthrough technique where the reference images are sparsely sampled along a path. The technique relies on a minimal user interface for rapid modeling. Simple meshes are drawn to model and represent the underlying scene in each of the reference images. The meshes, consisting of only several polygons for each image are then registered by matching feature lines to form an aligned 3D model. To synthesize a novel view, two nearby reference images are back mapped onto their models by a projective texture-mapping. Since the simple mesh is a crude approximation to the real model in the scene, feature lines are used as aligning anchors to further register and blend two views together and form a final novel view. The simplicity of the modeling yields rapid, "home-made" image-based walkthroughs. We have successfully produced walkthroughs from a set of photographs to show the effectiveness of the technique.

Introduction

A recently developed technique called TIP (Tour Into the Picture) [1] provides a constrained navigation into a single image. In this technique, a simple underlying structure of the scene is constructed from a 2D spidery mesh drawn through user interaction and the camera can be shifted to a nearby location to render a novel view of the scene. This allows a virtual touring into the picture.

In this paper we present a technique to generate novel views from a sequence of reference images along a path through the scene. One camera is usually inside the previous camera viewing frustum, shifted further towards the scene. These sample views form our image-based model of the scene. The reference images are photographs taken by a regular camera without registering or calibrating the viewing positions. One immediate application for this technique is home entertainment. One can use a regular camera to take a sequence of shots when touring a landscape and reconstruct and view a continuous tour at a later time. A potential application to video compression is also possible. A video segment can be largely down-sampled for storage, from which the full animation can be reconstructed and even offer some degree of freedom for navigation.

Algorithm Overview

The input of our algorithm is a sequence of views along a path. To ease our explanation, we first simplify the situation as only two input images. The extension to three or four images is straight-forward.

Given two nearby views sampled along a path, as in Figure bellow the algorithm consists of the following steps:

• Modeling: Independently, create a simple geometry for two images by employing a modified spidery mesh. Compute two 3D feature lines from the user drawn matching feature lines in two images.
• Registration: Register two geometries by placing the second geometry in the world space of the first geometry model.
• Rendering and Blending: For each novel view, render two geometries by projective texture-mapping the original images onto the geometries. Meanwhile, project the feature lines in two novel views. Match corresponding feature lines of reference images by warping two intermediate novel views. Blend two novel views together by a pixel-to-pixel cross-dissolve.

Modeled image-1 with registration line on the floor	Modeled image-2 with registration line on the floor	Registered models
Novel view rendered from image-1 with feature lines projection	Novel view rendered from image-2 with feature lines projection	Final warped and blended view

Results

Here are a few movie sequences we have created using our algorithm, using 17 feature line.

Three input images of a palm tree scene: Palm trees 1 (600k), a movie generated from the second and the third images.
Palm trees 2 (2.4M), a movie generated from three images.

Two input images of a statue scene: Statue (1M), a movie generated from the above two images.