Sascha Kirch, one of our members, and a team from UNED and Volograms just published a new article with the title “RGB-D-Fusion: Image Conditioned Depth Diffusion of Humanoid Subjects” in IEEEAccess.

In this joint work, the team developed a multi-modal denoising probabilistic diffusion model to generate high-resolution depth maps from low resolution monocular RGB images of humanoid subjects.

Accurately representing the human body in 3D is a very active research field given its wide variety of applications. Most 3D reconstruction algorithms rely on depth maps, either coming from low-resolution consumer-level depth sensors, or from monocular depth estimation from standard images. While many modern frameworks use VAEs or GANs for monocular depth estimation, Sascha and the team leverage recent advances in the field of diffusion denoising probabilistic models.

They implement a multi-stage conditional diffusion model that first generates a low-resolution depth map conditioned on an image and then upsamples the depth map conditioned on a low-resolution RGB-D image. The team further introduces a novel augmentation technique, depth noise augmentation, to increase the robustness of our super-resolution model. Lastly, they show how their method performs on a wide variety of humans with different body types, clothing and poses.

More from Sascha:
LinkedIn: https://www.linkedin.com/in/sascha-kirch/
Website: https://sascha-kirch.github.io/