Open Journal Systems

A single-image human body reconstruction method combining normal recovery and frequency domain completion

Yanyan Li, Zhihao Yang, Weilong Peng, Meie Fang

Article ID: 2295
Vol 4, Issue 2, 2023

DOI: https://doi.org/10.54517/m.v4i2.2295
VIEWS - 170 (Abstract)

Abstract

Reconstructing a 3D human body from a single image is convenient and efficient, but it faces challenges when the face is heavily occluded or the person is wearing loose clothing. In this paper, we propose a frequency domain-based method for completing missing parts of the human body using manifold harmonics and frequency domain analysis. Our approach involves linear interpolation of the incomplete 3D human body obtained through normal integration. The interpolated points are then projected onto the appropriate dimension of the frequency domain space using manifold harmonic bases. Through Laplacian mesh editing, the interpolated points are replaced, resulting in a refined and complete 3D human body. Our method surpasses the limitations of template human bodies and marching cubes algorithms, enabling more detailed feature reconstruction. By locally completing the body in a low-dimensional frequency domain space, our method avoids over smoothing and bulging issues, effectively filling the missing regions while maintaining mesh smoothness consistency. Experimental results demonstrate the effectiveness and superiority of our frequency domain-based completion method for accurate and detailed 3D human body reconstruction from single images.

Keywords

human body reconstruction; normal integration; manifold harmonics; Laplacian mesh editing

Full Text:

PDF

References

1. Li Z, Oskarsson M, Heyden A. Detailed 3D human body reconstruction from multi-view images combining voxel super-resolution and learned implicit representation. Applied Intelligence 2022; 52(6): 6739–6759. doi: 10.1007/s10489-021-02783-8

2. Yu Z, Zhang L, Xu Y, et al. Multiview human body reconstruction from uncalibrated cameras. In: Proceedings of the 2022 Conference on Neural Information Processing Systems; 28 November–9 December 2022; New Orleans, LA, USA.

3. Li X. Multi-view canonical pose 3D human body reconstruction based on volumetric TSDF. In: Karlinsky L, Michaeli T, Nishino K (editors). Lecture Notes in Computer Science, Proceedings of Computer Vision—ECCV 2022 Workshops; 23–27 October 2022; Tel Aviv, Israel. Springer; 2023. Volume 13805, pp. 293–397. doi: 10.1007/978-3-031-25072-9_27

4. Choi H, Moon G, Park JK, Lee KM. Learning to estimate robust 3D human mesh from in-the-wild crowded scenes. In: Proceedings of the 2022 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 21–24 June 2022; New Orleans, LA, USA. pp. 1475–1484.

5. Jiang W, Kolotouros N, Pavlakos G, et al. Coherent reconstruction of multiple humans from a single image. In: Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 13–19 June 2020; Seattle, WA, USA. pp. 5579–5588.

6. Choi H, Moon G, Lee KM. Pose2mesh: Graph convolutional network for 3D human pose and mesh recovery from a 2D human pose. In: Vedaldi A, Bischof H, Brox T, Frahm JM (editors). Lecture Notes in Computer Science, Proceedings of Computer Vision—ECCV 2020; 23–28 August 2020; Glasgow, UK. Springer; 2020. Volume 12352, pp. 769–787. doi: 10.1007/978-3-030-58571-6_45

7. Kocabas M, Huang CHP, Hilliges O, Black MJ. PARE: Part attention regressor for 3D human body estimation. In: Proceedings of the 2020 IEEE/CVF International Conference on Computer Vision; 20–25 June 2020; Nashville, TN, USA. pp. 11127–11137.

8. Jiang B, Zhang J, Hong Y, et al. Bcnet: Learning body and cloth shape from a single image. In: Vedaldi A, Bischof H, Brox T, Frahm JM (editors). Lecture Notes in Computer Science, Proceedings of Computer Vision—ECCV 2020; 23–28 August 2020; Glasgow, UK. Springer; 2020. Volume 12365, pp. 18–35. doi: 10.1007/978-3-030-58565-5_2

9. Li Z, Yu T, Pan C, et al. Robust 3D self-portraits in seconds. In: Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 13–19 June 2020; Seattle, WA, USA. pp. 1344–1353.

10. Alldieck T, Magnor M, Bhatnagar BL, et al. Learning to reconstruct people in clothing from a single RGB camera. In: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 15–20 June 2019; Long Beach, CA, USA. pp. 1174–1186.

11. Zhang H, Tian Y, Zhou X, et al. PyMAF: 3D human pose and shape regression with pyramidal mesh alignment feedback loop. In: Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV); 10–17 October 2021; Montreal, QC, Canada. pp. 11446–11456.

12. Loper M, Mahmood N, Romero J, et al. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics 2015; 34(6): 248. doi: 10.1145/2816795.2818013

13. Alldieck T, Pons-Moll G, Theobalt C, Magnor M. Tex2Shape: Detailed full human body geometry from a single image. In: Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV); 27 October–2 November 2019; Seoul, Korea. pp. 2293–2303.

14. Dong Z, Guo C, Song J, et al. PINA: Learning a personalized implicit neural avatar from a single RGB-D video sequence. In: Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 18–24 June 2022; New Orleans, LA, USA. pp. 20470–20480.

15. Li R, Xiu Y, Saito S, et al. Monocular real-time volumetric performance capture. In: Vedaldi A, Bischof H, Brox T, Frahm JM (editors). Lecture Notes in Computer Science, Proceedings of Computer Vision—ECCV 2020; 23–28 August 2020; Glasgow, UK. Springer; 2020. Volume 12368, pp. 49–67. doi: 10.1007/978-3-030-58592-1_4

16. Zheng Z, Yu T, Liu Y, Dai Q. PaMIR: Parametric model-conditioned implicit representation for image-based human reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence 2022; 44(6): 3170–3184. doi: 10.1109/TPAMI.2021.3050505

17. Saito S, Huang Z, Natsume R, et al. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In: Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV); 27 October–2 November 2019; Seoul, Korea. pp. 2304–2314.

18. Saito S, Simon T, Saragih J, Joo H. PIFuHD: Multi-level pixel-aligned implicit function for high-resolution 3D human digitization. In: Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 13–19 June 2020; Seattle, WA, USA. pp. 84–93.

19. Park JJ, Florence P, Straub J, et al. Deepsdf: Learning continuous signed distance functions for shape representation. In: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 15–20 June 2019; Long Beach, CA, USA. pp. 165–174.

20. Erler P, Guerrero P, Ohrhallinger S, et al. Points2Surf learning implicit surfaces from point clouds. In: Vedaldi A, Bischof H, Brox T, Frahm JM (editors). Lecture Notes in Computer Science, Proceedings of Computer Vision—ECCV 2020; 23–28 August 2020; Glasgow, UK. Springer; 2020. Volume 12350, pp. 108–124. doi: 10.1007/978-3-030-58558-7_7

21. Ren S, Hou J, Chen X, et al. Geoudf: Surface reconstruction from 3D point clouds via geometry-guided distance representation. In: Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV); 2–3 October 2023; Paris, France. pp. 14214–14224.

22. Pavlakos G, Choutas V, Ghorbani N, et al. Expressive body capture: 3D hands, face, and body from a single image. In: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 15–20 June 2019; Long Beach, CA, USA. pp. 10975–10985.

23. Zanfir M, Zanfir A, Bazavan EG, et al. Thundr: Transformer-based 3D human reconstruction with markers. In: Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV); 10–17 October 2021; Montreal, QC, Canada. pp. 12971–12980.

24. Vallet B, Lévy B. Spectral geometry processing with manifold harmonics. Computer Graphics Forum 2008; 27(2): 251–260. doi: 10.1111/j.1467-8659.2008.01122.x

25. Xiu Y, Yang J, Tzionas D, et al. ICON: Implicit clothed humans obtained from normals. In: Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 18–24 June 2022; New Orleans, LA, USA. pp. 13286–13296. doi: 10.1109/CVPR52688.2022.01294

26. Bhatnagar BL, Sminchisescu C, Theobalt C, et al. Combining implicit function learning and parametric models for 3D human reconstruction. In: Vedaldi A, Bischof H, Brox T, Frahm JM (editors). Lecture Notes in Computer Science, Proceedings of Computer Vision—ECCV 2020; 23–28 August 2020; Glasgow, UK. Springer; 2020. Volume 12347, pp. 311–329. doi: 10.1007/978-3-030-58536-5_19

27. Zheng Y, Shao R, Zhang Y, et al. DeepMultiCap: Performance capture of multiple characters using sparse multiview cameras. In: Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV); 10–17 October 2021; Montreal, QC, Canada. pp. 6239–6249.

28. Huang Z, Xu Y, Lassner C, et al. Arch: Animatable reconstruction of clothed humans. In: Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 13–19 June 2020; Seattle, WA, USA. pp. 3093–3102.

29. He T, Xu Y, Saito S, et al. Arch++: Animation-ready clothed human reconstruction revisited. In: Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV); 10–17 October 2021; Montreal, QC, Canada. pp. 11046–11056.

30. Biggs B, Novotny D, Ehrhardt S, et al. 3D multibodies: Fitting sets of plausible 3D human models to ambiguous image data. In: Proceedings of 2020 Conference on Neural Information Processing Systems; 6–12 December 2020.

31. Kolotouros N, Pavlakos G, Jayaraman D, Daniilidis K. Probabilistic modeling for human mesh recovery. In: Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV); 10–17 October 2021; Montreal, QC, Canada. pp. 11605–11614.

32. Wehrbein T, Rudolph M, Rosenhahn B, Wandt B. Probabilistic monocular 3D human pose estimation with normalizing flows. In: Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV); 10–17 October 2021; Montreal, QC, Canada. pp. 11199–11208.

33. Gabeur V, Franco JS, Martin X, et al. Moulding humans: Non-parametric 3D human shape estimation from single images. In: Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV); 27 October–2 November 2019; Seoul, Korea. pp. 2232–2241.

34. Xiu Y, Yang J, Cao X, et al. ECON: Explicit clothed humans optimized via normal integration. In: Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 17–24 June 2023; Vancouver, BC, Canada. pp. 512–523.

35. Chibane J, Alldieck T, Pons-Moll G. Implicit functions in feature space for 3D shape reconstruction and completion. In: Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 13–19 June 2020; Seattle, WA, USA. pp. 6970–6981.

36. Taubin G. A signal processing approach to fair surface design. In: Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques; 6–11 August 1995; Los Angeles, California, USA. pp. 351–358.

37. Kim H, Nam H, Kim J, et al. LaplacianFusion: Detailed 3D clothed-human body reconstruction. ACM Transactions on Graphics 2022; 41(6): 1–14. doi: 10.1145/3550454.3555511

38. Quéau Y, Durou JD, Aujol JF. Normal integration: A survey. Journal of Mathematical Imaging and Vision 2018; 60: 576–593. doi: 10.1007/s10851-017-0773-x

39. Yu T, Zheng Z, Guo K, et al. Function4D: Real-time human volumetric capture from very sparse consumer RGBD sensors. In: Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 19–25 June 2021; Online conference. pp. 5746–5756.

40. Smith D, Loper M, Hu X, et al. Facsimile: Fast and accurate scans from an image in less than a second. In: Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV); 27 October–2 November 2019; Seoul, Korea. pp. 5330–5339.

41. Cao X, Santo H, Shi B, Okura F. Bilateral normal integration. In: Karlinsky L, Michaeli T, Nishino K (editors). Lecture Notes in Computer Science, Proceedings of Computer Vision—ECCV 2022 Workshops; 23–27 October 2022; Tel Aviv, Israel. Springer; 2023. pp. 552–567. doi: 10.1007/978-3-031-19769-7_32

42. Ma Q, Yang J, Ranjan A, et al. Learning to dress 3D people in generative clothing. In: Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 13–19 June 2020; Seattle, WA, USA. pp. 6469–6478.

43. Fu J, Li S, Jiang Y, et al. StyleGAN-human: A data-centric odyssey of human generation. In: Karlinsky L, Michaeli T, Nishino K (editors). Lecture Notes in Computer Science, Proceedings of Computer Vision—ECCV 2022 Workshops; 23–27 October 2022; Tel Aviv, Israel. Springer; 2023. pp. 1–19. doi: 10.1007/978-3-031-19787-1_1