Mobile Graphics [Half-day Tutorial]

Marco Agus - CRS4 and KAUST
Enrico Gobbetti - CRS4 [Course Organizer]
Giovanni Pintore - CRS4
Pere-Pau Vázquez - Universitat Politècnica de Catalunya
Fabio Marton - CRS4

Abstract

The hardware for mobile devices, from smartphones and tablets to mobile cameras, continues to be one of the fastest-growing areas of the technology market. Not only mobile CPUs and GPUs are rapidly increasing in power, but a variety of high-quality visual and motion sensors are being embedded in mobile solutions. This, together with the increased availability of high-speed networks at lower prices, has opened the door to a variety of novel VR, AR, vision, and graphics applications. This half-day tutorial provides a technical introduction to the mobile graphics world spanning the hardware-software spectrum, and explores the state-of-the-art and key advances in specific application domains. The five key areas that will be presented are: 1) the evolution of mobile graphics capabilities; 2) the current trends in GPU hardware for mobile devices; 3) the main software development systems; 4) the scalable visualization of large scenes on mobile platforms; and, finally, 5) the use of mobile capture and data fusion for 3D acquisition and reconstruction.

This is an updated version of the tutorials that have been presented at Eurographics 2017 and SIGGRAPH Asia 2017. Besides a general update of all sections, novel material will be on large model rendering (with new content on volumetric data) and reconstruction (with the inclusion of recent indoor solutions). EG 2017 slides are available at http://vic.crs4.it/eg2017-tutorial-mobile-graphics/, while SIGGRAPH Asia 2017 slides are available at http://vic.crs4.it/sa2017-course-mobile-graphics/.

Tutorial Details

The outline of the individual sessions is detailed below.

Session 0: Introduction: General overview and plan of the course.
Session 1: Evolution of mobile graphics: Overview of mobile world, and its capillary diffusion; description of the main characteristics of mobile graphics terminals, with respect to network capabilities, graphics and displays, sensors and CPUs; discussion of current issues and future opportunities; analysis of the mobile application trends.
Session 2: Mobile graphics trends: General overview of rendering pipeline for interactive 3D mobile applications; evolution of 3D graphics applications, from remote rendering solutions to hybrid mobile/remote solutions, exploiting image-based or model-based methods; overview of current trends in hardware acceleration employing parallel pipelines, real-time ray tracing and multi-rate approaches: overview of other visual mobile applications like 3D acquisition and processing, physical simulations, and visual aberration correction.
Session 3: Graphics development for mobile systems: Overview of main operating systems employed in mobile platforms; description of currently used programming languages and tools and deployment environments; introduction to currently available 3D APIs.
Session 4: Scalable mobile visualization: Characteristics of mobile devices with respect to massive models and need for scalable technique with low CPU overhead; converting complex environments to image-based solutions; scaling in terms of geometry: comparison of general chunked triangulation/point-cloud methods with constrained techniques exploiting pre-defined compressed image formats and graphics primitives; improving visual quality with real-time screen-space based global illumination methods for mobile applications; approaches to support interactive volumetric exploration.
Session 5: Mobile metric capture and reconstruction: Overview of the mobile device sensor and sensor-fusion capabilities available on mobile devices, ranging from commodity smartphones and tablets to new generation spherical panoramic cameras (SPC); introduction to image-based 3D reconstruction methods running on mobile devices; mobile metric reconstruction methods using a combination of images and inertial acceleration data with an example implementation of a full acquisition and SfM reconstruction pipeline working in a limited bounding volume; real-world cases: mobile mapping and reconstruction of indoor scenes, from the limits of the perspective views to the advantages of the wide-FOV panoramic images; example of application: mobile reconstruction and exploration of indoor structures exploiting omnidirectional images; motion estimation and dense depth maps generation from small motion with mobile SPC; future trends: automatic mapping and reconstruction of indoor structures from 360 video sequences.
Session 6: Closing: Wrap-up and discussion.

Tutorial notes (slides)

Session 0: Introduction [pdf]
Session 1: Evolution of mobile graphics [pdf]
Session 2: Mobile graphics trends [pdf 2.1][pdf 2.2]
Session 3: Graphics development for mobile systems [pdf]
Session 4: Scalable mobile visualization [pdf 4.1/4.2/4.3/4.4] [pdf 4.5/4.6]
Session 5: Mobile metric capture and reconstruction [pdf]
Session 6: Closing [pdf]

Acknowledgments

This tutorial is partially supported by project TDM (RAS POR FESR 2014-2020, A1.2.2). We also acknowledge the contribution of the Spanish Ministerio de Economia y Competitivid, of FEDER (EU) funds under Grant No. TIN2017-88515-C2-1-R (GEN3DLIVE), of Sardinian Regional Authorities under projects VIGEC and VIDEOLAB, and of King Abdullah University of Science and Technology.