June 1998

Final project for CS224, May 1998.

Proposed by Frederic Leymarie and Daniel Sharvit

Teaser: Add some features to SKETCH to permit the interactive reconstruction of 3D objects from PERSPECTIVE views (you are in a bit of a rush? go download a PS file with some "real" examples ...!).

Interface issues

Perspective Drawing and its Geometry (concepts)

Dimensional Analysis through Perspective (implemented methods)

Bibliography

Our goals

Let us introduce some "perspective" in Sketch! Or if you prefer, let us extend TIP to handle not only Single-Point Perspective, but the more classical Two-Point and the "photogrammetric" Three-Point Perspective problems.

The usefulness of this ? Think of an educational tool to learn of drawing techniques from the Ancient Greeks, to the Masters of the Renaissance, Leonardo, Durer, Vermeer, Canaletto, Michelangelo, ... and (some!) Masters of our Century, Escher, Van Gogh, ...

But, what about being able to "naturally", through the gesture-driven concepts of Sketch, retrieve the geometry of a scene seen in perspective: extracts geometric 3D shapes.

On this journey, we shall learn about not only about drawing from Great Painters, but also about geometry from some famous and infamous Mathematicians: Euclid, Desargues, Chasles, ...

And finally, we hope to get some insight in our own visual sense of perception, and understand how architects such as Palladio can fool the eye of the naive, and why when you look at a (flat) postcard or canevas, the human (or is it cylopean?!) eye gets a strong sense of depth.

Story board - DRAFT (as of March 11, 1998)

Imagine you are given some digitized (or retinotopic) images, say a scanned postcard giving a view of NYC from the 5th avenue, the Pentagon, or instead, some famous painting that uses perspective effects, say "The music lesson" of J.Vermeer (1660).

What is needed in Sketch (JOT actually).

Some visualisation tools:

• Permit to overlay an image/view (grey-level or color) over the "ground plane" of SKETCH.
• Be able to manipulate the ground plane in 3D (rotations and translations).
• Be able to manipulate the Image: zoom in-out, overlay edges.

Retrieving "Calibration": where is the camera/eye ? what is the metric?

Interactively modify the geometry from orthographic to perspective by letting the user find/specify vanishing points (say, the user manipulates parallel lines (the grid of the ground plane), which "meets at infinity/horizon").

We will explore different levels of "perspectiveness":

• One vanising point: so-called "Parallel perspective"; useful for some Renaissance paintings, or street level horizontal views (a la TIP).
• Two vanishing points: so-called "Angular perspective", used by Durer in particular, good for a "corner" view at a house for example.
• Three vanishing points: so-called "Oblique perspective", good for ... aerial views! a view from above the Pentagon say.

Note that from the vanishing points, one can compute the camera parameters (i.e., position in 3D space, camera center, focal) ... if the user specifies some metric reference (say the approximate height of a door).

Extracting 3D Primitives

Permit to display wire-frame-like rendering of the object primitves of SKETCH, so that when the user "sketches" the geometric objects of the scene, she keeps seeing the overlayed image.

Thus we could go even further and now display other views of our reconstructed 3D scene (and maybe texture some of the surface with the overlayed image).

If we allow for multiple (generic) views to be available as inputs, say 4 or 5 views "around" a house (with a view from above). Then we could extend SKETCH to handle the additional information. At first the user works on each view separately like above. Once the camera calibration has been resolved interactively, the geometric objects extracted from one view can be re-projected in the others (this may provide for a good way of interactively refining the perspective geometry, if errors are visible, thanks to transparency).

Other "Calibration" methods

There are other geometric methods of deriving the calibration, which allow for user interaction (drawing figures, like ellipses, triangles, pyramids, ...). These could also be introduced in SKETCH.

This is in particular used if we also have access to a cartographic map, or a floor plan (architecture), where the user selects points, or lines in the image and corresponding entities on the map (so-called "method of the capable pyramid" ...pardon my French).

We can alternatively use Chasles Theorem, where instead the user draws a planar conic (on the map/plan) corresponding to some special feature points (at the "horizon") in the "image/view/postcard").

Summary

In summary, the idea is to use SKETCH gesture-based interface to first "draw" the perspective (in some sense, deforming the pencils of the 3D orthogonal coordinate grid used in SKETCH, to match the perspectiveness of the observed scene: 1, 2 or 3 Vanishing points/horizons).

For an example (or two), download this compressed PostScript file (2.6 M) .

(Near) Future Steps

The above has also the interest that, latter on, we could bring in some Computer Vision machinery to "help" the user, like for determining vanishing points, or "refining" the primitve models available in SKETCH, once the user as "sketched" the scene.

One can go back to the input image augmented with overlayed "simple" extracted geometric objects, and use it as an initialisation state for some model matching method (e.g. relying on high ordre polynomial fits, or deformable models), to obtain a finer fit, with a free-form model say.