CVPR 2023 Tutorial on
Automatic 3D modeling of indoor structures from panoramic imagery
Time and venue
- Date and time
- Monday, June 19th, 2023
- Venue
- Vancouver Convention Center, Vancouver, Canada
Overview
Summary. Creating and exploiting high-level 3D models of real-world indoor scenes from captured data are fundamental tasks with important applications in many fields. In this context, 360-degree capture and processing is very appealing, since panoramic imaging provides the quickest and most complete per-image coverage and is supported by a wide variety of professional and consumer capture devices. Research on inferring 3D indoor models from 360-degree images has been thriving in recent years, and has led to a variety of very effective solutions. Given the complexity and variability of interior environments, and the need to cope with noisy and incomplete captured data, many open research problems still remain. In this tutorial, we provide an up-to-date integrative view of the field. After introducing a characterization of input sources, we define the structure of output models, the priors exploited to bridge the gap between imperfect input and desired output, and the main characteristics of geometry reasoning and data-driven approaches. We then identify and discuss the main sub-problems in indoor reconstruction from panoramas and review and analyze state-of-the-art solutions for floor plan segmentation, bounding surfaces reconstruction, integrated model computation, and visual representation generation. We finally point out relevant research issues and analyze research trends.
Background and intended audience. The tutorial is at the intermediate level. Basic computer-vision and deep learning background is a pre-requisite. The target audience includes graduate students and researchers in 3D modeling and scene understanding, as well as practitioners in the relevant application fields. Researchers will find a structured overview of the field, which organizes the various problems and existing solutions, classifies the existing literature, and indicates challenging open problems. Domain experts will, in turn, find a presentation of the areas where automated methods are already mature enough to be ported into practice, as well as an analysis of the kind of indoor environments that still pose major challenges
Schedule
| Time | Duration | Lecturer | Topic | Sub-topics | Link |
|---|---|---|---|---|---|
| 09:00AM-09:05AM | 5' | Enrico Gobbetti | Opening | Course outline; Presenters introduction | Slides |
| 09:05AM-09:25AM | 20' | Enrico Gobbetti | Indoor capture, modeling and exploration basics | Definition and Application; Tasks and model; Data Capture; Artifacts; Reconstruction priors; Open Research Data; | Slides |
| 09:25AM-10:15AM | 50' | Giovanni Pintore | Room modeling | Bounding surfaces, exploiting priors, deep learning solutions | Slides |
| 10:15AM-10:30AM | 15' | - | BREAK | - | - |
| 10:30AM-11:15AM | 45' | Giovanni Pintore | Integrated indoor model computation | Multi-rooms; Ensuring consistency; Finding and modeling connections | Slides |
| 11:15AM-12:00PM | 45' | Marco Agus | Visual representation generation and exploration | Beyond geometric reconstruction; Appearance; panoramic exploration | Slides |
| 12:00PM-12:15PM | 15' | Enrico Gobbetti | Wrap-up and discussion | Summary of techniques and assessment of capabilities; Open problems | Slides |
| 12:15PM-12:30PM | 15' | ALL | Q&A | - | - |
Organizers and Lecturers
Giovanni Pintore
CRS4, Italy
Giovanni Pintore is a senior research engineer at the CRS4 research center in Italy. In his career he has coordinated and managed several international research and industrial projects in various fields, from space exploration to security management in urban environments. His research, widely published in major journals and conferences, spans many areas of computer graphics and computer vision, including deep learning architectures, geometry reasoning, panoramic scene understanding, multiresolution representations of large and complex 3D models, 3D multi-view reconstruction, and new generation mobile graphics. He regularly serves the scientific community through participation in conference committees and executive boards. His primary research focus is now in 3D reconstruction and immersive exploration of structured indoor scenes from omnidirectional images, on whose topic he has recently published papers at ISMAR, ECCV, CVPR, and CGF, and given courses at SIGGRAPH, SIGGRAPH Asia and 3DV in recent years.
Marco Agus
HBKU, Qatar
Marco Agus is currently Assistant Professor at Hamad Bin Khalifa University (HBKU) - Qatar Foundation in Doha, Qatar. He was previously research engineer at King Abdullah University of Science and Technology (KAUST), in Jeddah, Saudi Arabia and research scientist at Center of Research, Development and Advanced Studies (CRS4), in Cagliari, Italy. He obtained M.Sc. and Ph.D. from University of Cagliari, Italy. His research interests span different domains in visual computing, from haptics and visual rendering for medical applications, to real time exploration of massive models, to machine learning methods for electron microscopy biology data and indoor environments. He published more than 50 peer-reviewed papers on these topics. He taught courses at several important visual computing venues, including 3DV, ACM SIGGRAPH and Eurographics, and he regularly acts as committee member, reviewer, chair and associate editor for top journals and conferences in the visual computing domain.
Enrico Gobbetti
CRS4, Italy
Enrico Gobbetti is the director of Visual and Data-intensive Computing (ViDiC) at the Center for Advanced Studies, Research, and Development in Sardinia (CRS4), Italy. He holds an Engineering degree (1989) and a Ph.D. degree (1993) in Computer Science from the Swiss Federal Institute of Technology in Lausanne (EPFL), as well as Full Professor Habilitations in Computer Science and Information Processing from the Italian Ministry of University and Research. Prior to joining CRS4, he held positions at EPFL (Switzerland), UMBC (USA), and NASA/CESDIS (USA). At CRS4, Enrico develops and manages a research program in visual and data-intensive computing supported through institutional, industrial and government grants, including many national and international collaborative projects. His research spans many areas of visual and data-intensive computing and is widely published in major journals and conferences. The primary focus is the creation of innovative solutions for the acquisition, creation, processing, distribution and exploration of complex and/or massive datasets and real-world objects and environments. He regularly serves the scientific community through participation in editorial boards, conference committees, working groups and steering boards, as well as through the organization and chairing of conferences. He is a Fellow of the Eurographics Association.
Additional material
The course builds on our survey [PGF2020a], and on our tutorial presented at SIGGRAPH 2020 [PGF2020b]. While those articles and presentations covered many kinds of visual and non-visual input sources, we focus here on panoramic imagery. The topic has attracted a large interest in the community, as demonstrated by the growing number of papers in the latest computer vision conferences (see references in [PGF2020a]), and by the events that are regularly featured to discuss the latest advances (e.g., workshop on Holistic Structures for 3D Vision (ICCV, ECCV), OmniCV: Omnidirectional Computer Vision in Research and Industry (CVPR), ScanNet Indoor Scene Understanding Challenge (CVPR)). Our course fills a gap in the computer vision community, as the topic has not been the subject of tutorials presented at CVPR, ICCV, ECCV.
A copy of the slides for this course are available through the links in the "Schedule" table.
- [PGF2020a]
- Giovanni Pintore, Claudio Mura, Fabio Ganovelli, Lizeth Fuentes-Perez, Renato Pajarola, and Enrico Gobbetti. State-of-the-art in Automatic 3D Reconstruction of Structured Indoor Environments. Computer Graphics Forum, 39(2): 667-699, 2020. DOI: 10.1111/cgf.14021
- [PGF2020b]
- Giovanni Pintore, Claudio Mura, Fabio Ganovelli, Lizeth Fuentes-Perez, Renato Pajarola, and Enrico Gobbetti. Automatic 3D Reconstruction of Structured Indoor Environments. In SIGGRAPH 2020 Courses. Pages 10:1-10:218, August 2020. DOI: 10.1145/3388769.3407469
BibTeX
@misc{PAG:2023:CTA,
author = {Giovanni Pintore and Marco Agus and Enrico Gobbetti},
title = {CVPR2023 Tutorial on Automatic 3D modeling of indoor structures from panoramic imagery},
howpublished = {\url{http://vic.crs4.it/vic/cvpr2023-tutorial-pano}},
year = {2023}
}
Acknowledgments
We acknowledge the contribution of NPRP grant #0403-210132 AIN2: Artificial Intelligence for Indoor Digital Twins from the Qatar National Research Fund (a member of Qatar Foundation).