Seamless image-based texture atlases using multi-band blending Cédric Allène

Jean-Philippe Pons

Renaud Keriven

Université Paris-Est – Ecole des Ponts – CERTIS

Seamless image-based 3D textures

Outline 1) Context: 3D reconstruction 2) Image regions choice 3) Multi-view blending 4) Atlas building and results

Seamless image-based 3D textures

- 1 Context: 3D reconstruction

Seamless image-based 3D textures

1 – Context: 3D reconstruction

3D reconstruction from multi-view

First step: Get a set of pictures or the model to reconstruct.

Seamless image-based 3D textures

1 – Context: 3D reconstruction

3D reconstruction from multi-view

Second step: Autocalibration from your set of pictures.

Seamless image-based 3D textures

1 – Context: 3D reconstruction

3D reconstruction from multi-view

Third step: 3D mesh building.

Seamless image-based 3D textures

1 – Context: 3D reconstruction

3D reconstruction from multi-view

Fourth step: Add colored textures on the mesh from multi-view images.

Seamless image-based 3D textures

1 – Context: 3D reconstruction

3D reconstruction from multi-view Problems: ‰ Which image should the texture be from? ‰ How to minimize color discontinuities at the frontiers of regions from different images? This is here our method comes into play!

Fourth step: Add colored textures on the mesh from multi-view images.

Seamless image-based 3D textures

- 2 Image regions choice

Seamless image-based 3D textures

Surface partition 2 – Image regions choice

We note: ‰ ‰

the input calibrated images, and the projection from 3D space to image

We have to assign each face of the mesh in to one of the input views in which it is visible. We obtain a labeling vector What do we want? ‰ good visual detail, and ‰ color continuity at region boundaries

Seamless image-based 3D textures

Energy terms 2 – Image regions choice

Good visual detail:

Color continuity at region boundaries:

where neighbour faces

is a dissimilarity term between the and with labels and .

Seamless image-based 3D textures

Energy minimization 2 – Image regions choice

Global energy to minimize:

Since is a sum of regular functions, we can use the MRF/min-cut minimization method to solve our problem. Problem: Even if minimized, color discontinuities remain between regions of different labels…

Seamless image-based 3D textures

- 3 Multi-view blending

Seamless image-based 3D textures

3 – Multi-view blending

Multi-frequencies blending Method proposed in: P. J. Burt and E. H. Adelson. A multiresolution spline with application to image mosaics. ACM Trans. on Graphics, 2(4), 1983.

Two main functions: ‰ Reduce function (RED): i i

‰

Input: image I

Output: image I’ which dimensions are half of those of input image I reduced through a gaussian kernel

Expand function (EXP): i i

Input: image I

Output: image I’ which dimensions are twice those of input image I expanded through the same gaussian kernel

Seamless image-based 3D textures

Gaussian & laplacian pyramids 3 – Multi-view blending

‰ Gaussian

pyramid (left), obtained by successive reductions of input image.

‰ Laplacian

pyramid (right), obtained by substractions of two gaussian pyramid levels.

As a consequence, the laplacian pyramid is a multi-frequencies decomposition.

Seamless image-based 3D textures

3 – Multi-view blending

Summing laplacian pyramid

Important remark: Summing through successive expansions the laplacian pyramid allows to get back the gaussian pyramid and, so, the original image.

Seamless image-based 3D textures

3 – Multi-view blending

2D blending

? 2D blending: Merging in a “natural” way the pictures following their respective masks. First step: Build the laplacian pyramid for each image, and Build the gaussian pyramid for each mask.

Seamless image-based 3D textures

3 – Multi-view blending

2D blending

Second step: Create a new laplacian pyramid from the image ones through the gaussian pyramids of their masks.

Seamless image-based 3D textures

3 – Multi-view blending

2D blending

Third step: Get the final blended image by summing the newly created laplacian pyramid.

Seamless image-based 3D textures

Multi-view blending 3 – Multi-view blending

Straightforward from 2D blending using the projection functions for each image . So we obtain the final color at point x of the surface:

with

Seamless image-based 3D textures

- 4 Atlas building and results

Seamless image-based 3D textures

4 – Atlas building and results

Results – Aiguille du Midi (France)

"Naive" approach

Optimized patchwork

Optimized patchwork + blending Copyright Bernard Vallet (www.bvallet.com)

Seamless image-based 3D textures

4 – Atlas building and results

Atlas – Aiguille du Midi (France)

Seamless image-based 3D textures

4 – Atlas building and results

Results – Ettlingen castle (Germany)

"Naive" approach

Optimized patchwork

Optimized patchwork + blending Courtesy Christoph Strecha, EPFL (http://cvlab.epfl.ch/strecha/multiview/)

Seamless image-based 3D textures

4 – Atlas building and results

Atlas – Ettlingen castle (Germany)

Seamless image-based 3D textures

4 – Atlas building and results

Conclusion Seamless image-based 3D textures: ‰ Graph-based combinatorial optimization for texture partition ‰ Multi-view blending on the partition

Advantages: ‰ Good results Drawbacks: ‰ Computing time dependant of the number of levels in the pyramids

Future work: ‰ GPU acceleration of the blending part

Seamless image-based 3D textures

Thank you for your attention! Any question?