Figure 1 : Multimedia application domains for new IT.
3rd
European
Digital Cities Conference
December 1 & 2, 1997
Berlin, Germany
Keywords:
3D geo-referenced multimedia datasets, architecture, augmented reality, built environment, cultural heritage, hypermaps, photo-texture, realistic/faithful simulation, urban databases, virtual reality, virtual tourism.
The part of the world population living in cities keeps growing. Within the next 20 years, it is expected that more than 50% of the people will live in densely populated urban areas (close to 75% in industrialised countries). The associated continuous growth of the « built environment » (buildings, houses, road and energy networks, transit systems, etc.) should benefit from new Information Technologies (IT) and in particular Virtual and Augmented Realities, i.e., the making of virtual 3D spaces and objects that mimics or complement reality as we normally apprehend it through our sense of vision. Such virtual scenery shall permit to better plan, design, engineer and deliver the needed evolutions of the built environment. Noticeably, the present day economical impact of construction of the built environment is formidable: it represents close to half of the capital investment in most of the Member States of the European Union.
The 3D digital reconstruction of the built environment will benefit from advances made in the (semi-)automatic exploitation of image data (such as aerial and ground-based video recordings). Such imagery can be used not solely to recover the geometry of the built environment but also the (photo-)textures of the surfaces making this scenery. The realistic and faithful 3D models thus recovered will find application in diverse fields such as architecture, urban planning, video games and simulators. Furthermore, once this correspondence between geometry and photometry is established, the modelling can be made even more sophisticated by recovering material properties such as electromagnetic reflectance functions. This shall open-up the door to applications in fields such as cellular phone communication planning in urban areas, or illumination modelling of buildings.
In this presentation we will discuss potential applications of virtual reality via the introduction of two complementary interactive multimedia information systems we call CyberMonument and CyberCity:
=> A CyberMonument provides access and use of tools for authoring and distributing multi-dimensional and multimedia databases of interiors and exteriors of (famous) architectural sites such as historical buildings, monuments, cathedrals, etc.
=> A CyberCity provides authoring and database management tools for photo-realistic visits and tele-use of large and complex 3D urban multimedia databases in a pan-European telematics context.
The new IT required to produce CyberMonuments and CyberCities are now being developed by leaders of the industry and academic worlds. We have already achieved some prototype demonstrations which we illustrate in this paper. The application of such new tools in the context of VR or AR shall permit in particular to better tackle the growing complexities of urban planning in the European metropolis of tomorrow.
Early in the next century most, if not all, Human-Computer Interactions (HCI) will involve some degree of Virtual Reality (VR) or Augmented Reality (AR) through multimodal interfaces (3D dynamic vision, 3D sound, vocal recognition, etc.). Of these new interactive modalities, the need for realistic and faithful 3D rendition of existing scenery, possibly augmented with some artificial objects, animated or not, is already emerging from different fields: architecture, electronic publishing, training, urban planning, virtual tourism, etc. At the moment, the industry still awaits « light » systems that will permit to efficiently produce such 3D realistic and faithful models. Noticeably, many VR companies, mostly involved in producing computer systems permitting immersive HCI, are having difficulty to see their market grow. We are convinced that the capacity to efficiently produce 3D scenery will be a major factor in boosting the VR European market growth potential.
One of the most spectacular applications of VR/AR concerns the possibility of presenting realistic architectural structures, projects or designs. If faithfulness is added by inserting a new design in an existing setting, such VR/AR application reproduce as no other medium the sense of space which is essential for the understanding of architecture and its impact on the surrounding environment. In this respect virtual visits can give new presentation standards for architecture and can also make attractive and understandable architecture to non specialists, i.e., to people unable to read technical drawings. Furthermore, realistic and faithful 3D models will permit to better « memorise » famous monuments, which shall prove invaluable in case of eventual partial or complete destruction (due e.g. to earthquake, fire, flood, war, ...). Moreover, archaeologists already require sophisticated IT tools to be able to reconstruct ancient sites, which vanished hundreds of years ago, on the basis of available information gathered at excavations.
Electronic publishing is in its infancy. Most of the products at this time are electronic duplications of paper books. The best use of the potential of the electronic medium needs further investigation. Virtual visits are a new form of interactive imagery available only via computers and not through books or conventional audio-visual presentations. Consider, for example, a « 3D Yellow Pages » kind of electronic handbook, where an existing commercial street will be visited "virtually" by a potential buyer in search of a store that answer her/his needs. A hotel room may also first be visited "virtually" before booking it, maybe in function of the available view or the furniture. The fields of education, communication, building maintenance and cartography will also greatly benefit from such new means of (re-)presenting information about the built environment.
Thus far, the dramatic data acquisition costs, the use of large graphics workstations, the massive amounts of memory necessary in order to host photorealistic detail, have made virtual visits of architecture un-affordable to the field of electronic publishing. On the other hand, authoring tools making convenient & affordable the fulfilment of virtual visits of multimedia architectural structures (made of geometric, photo-textural, video, etc., elements) open new scenarii in the industry of electronic publishing which is currently based on the rather limited resources of PCs, CD-ROM's & present-day Internet.
Computer Based Training (CBT), for police, rescue and military forces, for maintenance teams working in hazardous environments (e.g., nuclear stations), etc., requires to feed the simulators with realistic, faithful and up-to-date 3D DB integrating explicit models of the exteriors and interiors of buildings or infrastructures where actions shall eventually take place. Such databases can also be used for tele-presence activities, like the navigation and control of robots and tele-manipulators in adverse conditions or hazardous sites. Think also of AR applications for fire-fighters who could benefit from accurate visual information when heavy smoke clutters their normal vision.
At the moment, the production of such 3D DB is done mostly manually, if it is done at all: in fact, simpler 3D DB, only grossly mimicking the reality, are produced at a high cost (because it requires high amounts of time, memory and specialised human resources).
The principal targeted market segments are the following (in alphabetical order):
Figure 1 : Multimedia application domains for new IT.
Noticeably, these different market segments can all benefit from the technologies developed within CyberMonument and CyberCity. Figure 1 gives a schematic view at the important domains of applications for the emerging technologies used to generate & manage 3D geo-referenced[2] datasets augmented with other multimedia data. The technologies thereby developed will become available & useful for all these domains. The concept of a CyberMonument or CyberCity provides for specific applications of these technologies, demonstrating their feasibility and potential usefulness.
Virtual presentations - realistic & faithful - of existing objects and sites are finding an increasing number of applications, e.g. in the industries of entertainment and movie production (such as « Star tours » of Disney and Lucasfilm), or for training and simulation systems in the medical, military and industrial fields (such as for the simulators of TTS, GE, CAE, etc.). High implementation costs and expensive visualisation equipment are common to all current applications of virtual reality of convincing photo-realistic quality. As well, photo-realistic virtual visits of objects and places, in spite of their tremendous potential, remain un-affordable to the electronic publishing industry with today's technology. Indeed, electronic publishing is in its infancy; most of the products at this time are electronic duplications of paper books. Similarly, a technological alternative to traditional procedures for modelling existing architectural sites from technical drawing and images is needed. These procedures are extremely labour intensive and their results heavily rely on the skills of the human operator. In essence, the economical burden of current practice concerning photo-realistic virtual visits lies mostly in the following areas:
Therefore, at the heart of our concerns stands the development of techniques and tools which make the authoring of photo-realistic virtual visits of architectural sites more affordable to the fields of electronic publishing, architecture, simulation, entertainment, etc. Furthermore, we expect to make the developed authoring tools available on current, or soon to be released, low to mid range dedicated hardware technologies. The demand for such technologies already exist, for example from companies like Absolut (producer of the Absolut Museum software package to permit navigation within an art's gallery), Sega, Virtual World, Sony, Iwerks entertainment, MCA, Disney, ILM, Pixar, etc., which are all producing 3D datasets for their own needs in VR applications to cinema and entertainment, albeit using costly technologies.
The emerging needs for efficient urban information systems of large cities or municipalities should be unequivocal. In particular, we emphasise that:
Concurrently, some key technological advances are made:
Furthermore, other related needs and markets are also emerging, such as:
In order to address the needs of these emerging « markets », progress is mainly required in two related technological domains:
Note that the main source of input data comes from images: aerial or satellite imagery, video or still pictures, etc. Other sources of data include: maps and plans, existing 3D sets (e.g., from CAD applications), GPS data, calibration data, reflectance models, etc. Furthermore, in order to provide for useful applications, the reconstructed 3D scenery should integrate data of complementary nature, such as text, sound and recordings, as well as alternative but related visual data (videos, documentaries, post-cards, etc.), thus subsuming a real multimedia database (MM DB).
Noticeably, the geographical context provides a spatial reference framework, which permits to establish natural topological relationships between complex objects, such as buildings or infrastructures. Furthermore, a well-established ensemble of spatial analysis concepts for manipulating the data is therein provided. These are two fundamental characteristics that distinguish a GIS[3] from a classical DBMS[4], such as is used in CAD[5] packages found on the market for the manipulation of complex 3D datasets (e.g., in industrial design: cars, engines, pipeline networks, etc.).
CyberMonument and CyberCity are projects which exemplify the potential of new IT for applications within the domains of VR and AR. In the following we summarise the characteristics of these two (complementary) projects in the making and present some examples of reconstructed 3D models.
The main objective of this project is to provide access and use of tools for authoring and distributing multi-dimensional and multimedia databases of interiors and exteriors of famous architectural sites such as historical buildings, monuments, cathedrals, etc. In order to reach that goal, we are carrying through the following main technical objectives:
The visitor of our cyber-sites is able to interact with three-dimensional photo realistic textured digital models or « CyberMonuments », navigating toward the information structure of interest and interacting with it, e.g. by:
In short, we produce a description of the scene consisting in a list of objects, originating from an appropriately developed database of elementary architectural volumetric and texture models, each associated with an affine map which encodes their location in the environment. We also provide tools to access and integrate other related sources of information, i.e., audio, text, video, music, thus « subsuming » the information system to a truly multimedia system. Noticeably, the outputs of CyberMonument conform with existing norms or standards in the making, e.g. VRML[7], an emerging networking standard for manipulating and managing 3D multimedia objects.
The distribution of a reconstructed site, or CyberMonument, can be demonstrated using digital media, like CDI or DVD, as well as established visualisation tools (e.g. « SceneViewer » of Sillicon Graphics Inc.). This permits a 4D and fully interactive presentation of the site in time and space.
CyberMonument targets the development of techniques and tools which make the authoring of photo-realistic virtual visits of architectural sites more affordable to the fields of electronic publishing, architecture, simulation, entertainment, etc. Moreover, when looking at today's market of VR applications, the State of the Art relies upon « simple », albeit massive, image-based CD-ROM production permitting an access to large image databases, mimicking different a-priori fixed paths. Commercial products for real-time « interactive » visualisation of objects and places are indeed already available (e.g. « Quicktime VR » by Apple, « Panoramic View » by Microsoft). However, such products resort to cleverly managed databases of pre-stored images taken « in situ » according to complex, time consuming and costly procedures.
In fact, such purely image-based applications give little true interaction capacities to the user or « visitor », when compared to a full 3D textured model. Furthermore, little, if any, « creativity » is possible. The « director » cannot apprehend a site in order to build a scenario. The « tourist » cannot explore or fly-through the monument at his/her own will. The « architect » cannot modify the existing site to simulate new perspectives, new improvements,... As Bruno Zevi pointed out more than 40 years ago:[8]
|
« ... if the specific character of architecture is the inner space and its value comes from the possibility of visiting in a sequence all the spatial stages, it is clear that nor one neither a hundred photographs will ever represent a building in full. ... ». « Technical drawings, miniature models, photographs and movies, these are the tools at our disposal to represent spaces. [...] If architecture, as cubists believed, could be defined by four dimensions, we would have adequate tools to represent spaces. Architecture, [...] has dimensions beyond [the 3 spatial ones.] Movies may represent one, two, three of the possible paths of the viewer in space, but one learns about space only from infinite paths. Furthermore one thing is to see the actors moving on a stage laying back on a theatre chair, another is living and acting on the stage ... » |
Even when playing on the « same ground » - of CD-ROM production - the new IT we are developing enhance dramatically the quality/cost ratio of photo-realistic virtual visits. This shall nourish a wide range of multimedia applications for various fields, such as archaeological, architectural and historical sciences, as well as virtual tourism, education and entertainment which are currently waiting for a sensible lowering of the production costs of this type of realistic and faithful data.
The key idea to lower realisation costs of CD-ROM production is to acquire synthetic images from the digitally reconstructed photo-realistic site. Interactive real-time visualisation can then be achieved by using the best suited COTS[9]. Interacting with the 3D models, the user is able to chose the resolution level in concordance with the available resources, and to pre-compute the image database most practical. This method permits to achieve any desired LoD[10] without impairing real-time visualisation performance.
Finally, we note that CyberMonument represents a complement to another information system in the making which aims at producing and managing 3D scenes of exteriors at much larger scales. That is, the production of entire urban or rural sites - a project we nicknamed « CyberCity ».
CyberCity is a project devoted to the generation and tele-use of hypermedia 3D urban databases. CyberCity requires similar technologies as CyberMonument, with however some specific trends:
Principal end-user applications include:
To adequately support such demands, we have designed CyberCity in the framework of a distributed network of hypermedia servers, called « GeoServers ». A GeoServer permits local or remote access for clients or data providers, through a standardised user-friendly GUI, loadable over the network, e.g. as an « applet ». Data and added value services are made accessible through a flexible broadband transport technology, such as ATM[11], ensuring high transmission rates. A schematic view at the concept of GeoServer-based services for CyberCity is given in Figure 2. Finally, in Figures 3, 4 and 5 we present some pictures illustrating the CyberCity and CyberMonument projects.
Figure 2 : Synoptic view at GeoServer-based services for CyberCity.
(a) Reconstructed geometry seen as a wireframe.
(b) View at the photo-textured 3D model (interiors).
Figure 3 : CyberMonument: Example of a reconstructed site - the National Library of Vienna - (a) wireframe, (b) photo-textured interiors.
Figure 4 : Schematic view at CyberCity - from
« Object Oriented DBMS », to « Level
of Details (LoD) », to « test and
validation » (here for the city of Vienna).
Figure 5 : CyberCity: Example of a reconstructed urban site
- from geometry, to shaded rendition, to a photo-textured 3D model.
In this exposé we started by emphasising the growing needs for new tools to better represent and manipulate realistic 3D models of urban sites. The built environment represents the major part of capital investment in Europe. Cities keep growing in size and complexity. Simultaneously advances in IT, in particular through the rapid development of the Internet and the access to high resolution aerial and satellite imagery, is creating demand from new emerging markets such as electronic publishing for virtual tourism and retailing, hypermedia cartography, virtual studio for the cinema and game industries, computer based training, etc.
This lead us to conclude that advances in two related technological domains were required to be made in the near future. That is, we need to develop, firstly, tools permitting the efficient reconstruction of 3D scenery from useful source of data (mostly images), and, secondly, new database management tools for the exploitation of such reconstructed 3D geo-referenced multimedia datasets.
We further demonstrated our recent efforts in specific applications:
Such projects are aimed at uncovering the technical difficulties in going toward obtaining information systems permitting the efficient reconstruction and management of 3D multimedia geo-referenced datasets. Furthermore, within the course of conducting such projects we target to obtain feedback from users and provide them with a better understanding of the potentials of the developed technologies in order to satisfy the needs of new emerging markets and be better prepared to tackle the complexity of the urban planning of our future European metropolis.
[ Cortelazzo95Museum] G.M.Cortelazzo, R.Frezza & F.Pollastri, « Museums on the net: an Italian perspective », Proc. « Int. Conf. Science & Technology for the safeguard of the cultural heritage in the Mediterranean basin », Catania, November 1995.
[ Cortelazzo96Paint] G.M.Cortelazzo, R.Frezza & F.Visona, « An inexpensive method for electronically displaying paintings at predetermined resolution », « Proc. Electronics in the Visual Art », Florence, February 1996.
[ Cortelazzo97Photo] G.M.Cortelazzo, S.Sacco & F.Marcotto, « Photorealistic Virtual Visits of Historical Architecture via Image-Based Rendering », Proc. of « International Workshop on Synthetic-Natural Hybrid Coding and Three Dimensional Imaging » (IWSNHC3DI'97) Rhodes, Greece, Sept. 5-7, 1997.
[ Debevec96] P.E.Debevec, C.J.Taylor & J.Malik,
« Modelling and Rendering Architecture from Photographs: A
hybrid geometry- and image-based approach », Conf. Proc. of
SIGGRAPH '96, New-Orleans, USA, Aug. 1996, pp.11-20.
http://www.cs.berkeley.edu/~debevec/Research/
[ Florenzano96] M.Florenzano, J.-Y.Blaise & P.Drap,
« PAROS: Le sens de la mesure - Photogrammétrie
et modèles architecturaux », International
Archives of Photogrammetry & Remote Sensing (XVIII ISPRS
Congress), vol.XXXI, part B5, pp. 167-172, Vienna, July 1996.
http://www.gamsau.archi.fr/PAROS/PUBLIS/FORUM/ISPRS.htm
(French version)
http://www.gamsau.archi.fr/PAROS/PUBLIS/ISP_ENG/ISP_ENG.htm
(English)
[ Gruber96Extract] M.Gruber & M.Maresch « Extracting Geometry and Texture of Indoor Sites from Panoramic Images », in « Machine Perception Applications », A.Pinz & W.Pölzleitner eds., Proc. IAPR TC-8 Workshop, pp.239-249, Graz, Austria, Sept. 1996.
[ Gruber96Req] M.Gruber, F.Leberl & M.Maresch, « Requirements for Photorealistic 3D Modelling of Urban Areas », International Archives of Photogrammetry & Remote Sensing (XVIII ISPRS Congress), vol.XXXI, part B3, pp. 261-265, Vienna, July 1996.
[Jepson96] W.Jepson, R.Liggett & S.Friedman, « Virtual Modeling of Urban Environments », Presence, vol.5(1), pp.72-86, Winter 1996.
[ Kofler96] M.Kofler, H.Rehatschek, & M.Gruber, « A
Database for a 3D GIS for Urban Environments Supporting Photo-Realistic
Visualisation », International Archives of
Photogrammetry & Remote Sensing (XVIII ISPRS Congress),
vol.XXXI, part B2, pp. 198-202, Vienna, July 1996.
http://www.icg.tu-graz.ac.at/~kofler/isprs96/paper.html
[ Leymarie96Realise] F.Leymarie et al., « REALISE: Reconstruction of Reality from Images Sequences », IEEE Proc. of IEEE « International Conference on Image Processing » (ICIP'96), Vol.III, pp.651-654, P.Delogne ed., Lausanne, Switzerland, Sept. 1996.
[ Leymarie97GIS3D] F.Leymarie, « Exploitation of 3D Georeferenced Datasets in a GIS», Proc. International Workshop on « HPCN Exploitation of Multimedia Databases », Ispra, Italy, March 1997, pp.10-29. (European Commission - JRC, n° EUR 17349 EN).
[ Maltret96] J.-L.Maltret & J.Zoller, « Simulation of Architectural and Urban Morphology - From automated 3D Modelling to Virtual Reality: problems and challenges », in Proc. of the « Workshop on 3D - City Models », held in Bonn, Oct. 9-11, 1996.
[Ouhyoung96] M.Ouhyoung & Y.-h. Yan, « A Three-Dimensional Building Authoring Tool Based on Line Drawing Understanding », Presence, vol.5(1), pp.1-12, Winter 1996.
[ Palladio65] A. Palladio, « The four books of architecture », traduction Isaac Ware, Dover Pub., 1965.
[ Rockwood97] A.P.Rockwood & J.Winget, « Three-dimensional object reconstruction from two-dimensional images », Computer-Aided Design, Vol.29(4), pp.279-285, 1997.
[Schmitt95] G.Schmitt et al., « Toward Virtual Reality in Architecture: Concepts and Scenarios from the Architectural Space Laboratory », Presence, vol.4(3), pp.267-285, Summer 1995.
[ Streilhen94] A. Streilhen, « Towards automation in architectural photogrammetry: CAD-based 3D-feature extraction », ISPR Journal of Photogrammetry & Remote Sensing, vol.49(5), pp.4-15, 1994.
[ Tannous97Saphir] I.Tannous et al., « SAPHIR 3D: A System for 3D Site Model Acquisition », IAPRS, Vol.32, Part 3-4W2, « 3D Reconstruction and Modelling of Topographic Objects », Stuttgart, Sept. 1997.
[ Zevi56] B.Zevi, « Saper vedere
l'architettura », G.Einaudi ed., 1956 (extracts from
Chapter III on the « Representation of Space »).
[1]WWW: the World Wide Web or network of networks, also called the Internet.
[2] Data is geo-referenced if its associated coordinates and identity is set with respect to a reference model for the Earth's surface.
[3] GIS: Geographical Information System.
[4] DBMS: DataBase Management System.
[5] CAD: Computer Aided Design.
[6] Andrea Palladio: Famous Italian architect, 16th Century.
[7] VRML: Virtual Reality Modeling Language.
[8] Thanks to Prof. G.M.Cortelazzo of Padova University for this reference.
[9] COTS: Commercial Off-The Shelf softwares.
[10] LoD: Level of Detail; in the context of a multi-scale representation of data.
[11] ATM: Asynchronous Transfer Mode.
Page created & maintained by Frederic Fol Leymarie,
1998 (last update: July 2007).
Comments, suggestions, ...: ffl(at)gold(dot)ac(dot)uk