Real-time Reconstruction of Dynamic Scenes based on Voxel Hashing
Overview
This is the master thesis which I did on the topic of real-time reconstruction of dynamic scenes.
Problem Setting
Dynamic reconstruction is notoriously challenging, even to this day. The reason for this is that in contrast to static reconstruction, the scene can change over time, e.g. a person walks by or clothing moves in the wind. This leads to ambiquities in the reconstruction process, in other words too many unknowns like the position of the camera, the geometry of the scene and how the geometry changes over time.
Despite these challenges, some early approaches have tackled this setting with one particular work standing out: DynamicFusion [1]. Similar to his predecessor for static scenes, KinectFusion [2], this approach allowed the real-time reconstruction of dynamic scenes without the use of scene assumptions like templates.
Nonetheless, some problems which have been present and tackled in KinectFusion, are also existent in DynamicFusion. One such problem is the use of a fixed-size grid of voxels used for reconstruction, which limits the size and resolution with which the scene can be reconstructed. This particular problem has been solved for pipelines like KinectFusion through the use of voxel hashing [3].
Approach
In the scope of this project, the problem of the fixed-sized voxel grid, used in DynamicFusion, was tackled and solved through the use of voxel hashing.
Due to the additions made to the KinectFusion pipeline in order to reconstruct dynamic scenes, the task of incorporating voxel hashing into the new pipeline is not as straightforward as it may seem at first glance, which provided the challenging aspect of this project.
The result of this project was a re-implementation of DynamicFusion based on voxel hashing.
Used Technologies
This project was implemented in C++ and based on an existing framework called InfiniTAM [4]. Due to the real-time aspect, multiple parts of the algorithm were implemented to run on the GPU through the use of CUDA.
References
- Newcombe, Richard A., Dieter Fox, and Steven M. Seitz. "Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
- Izadi, Shahram, et al. "Kinectfusion: real-time dynamic 3d surface reconstruction and interaction." ACM SIGGRAPH 2011 Talks. 2011. 1-1.
- Nießner, Matthias, et al. "Real-time 3D reconstruction at scale using voxel hashing." ACM Transactions on Graphics (ToG) 32.6 (2013): 1-11.
- Kähler, Olaf, et al. "Very high frame rate volumetric integration of depth images on mobile devices." IEEE transactions on visualization and computer graphics 21.11 (2015): 1241-1250.