Open Journal Systems

Automatic labeling of 3D facial acupoint landmarks

Junjie Yin, Meie Fang, Weiyin Ma

Article ID: 2476
Vol 5, Issue 1, 2024

DOI: https://doi.org/10.54517/m.v5i1.2476
VIEWS - 46 (Abstract)

Abstract

As special marks on a human face, facial landmarks reflect the facial features of various parts of the face, which is crucial in biomedicine and medical imaging. In addition, facial landmarks are also important features in computer vision such as face detection, face recognition, facial pose estimation, and facial animation. In this paper, we construct a 3D facial acupoint annotated dataset by labeling 37 facial acupoints on 846 neutral face triangle mesh on the FaceScape dataset. Based on these annotated data, we use a feature template matching method to realize the automatic annotation of 37 acupoints on triangle meshes. We used 40 meshes as the training set to extract the geometric patterns of 3D acupoints and then measured the performance of the automatic labeling algorithm on 20 meshes and 806 meshes as the test sets. In the training process, we extract the tangent plane for each landmark, project the neighbor vertices of the landmark to the tangent plane, and construct the feature image with R × R resolution through the bounding box of the projected points. In the testing process, we use the pattern images extracted during training to find the average features and use them as a guide to optimize the predicted landmarks. The experimental results show that our automatic acupoint labeling method has achieved good results.

Keywords

landmark detection; local geometric feature; acupoints; dataset

Full Text:

PDF

References

1. Vrochidou E, Papić V, Kalampokas T, Papakostas GA. Automatic Facial Palsy Detection—From Mathematical Modeling to Deep Learning. Axioms. 2023; 12(12): 1091. doi: 10.3390/axioms12121091

2. Kim S, An H, Song M, Lee S, et al. Automated Marker-Less Patient-to-Preoperative Medical Image Registration Approach Using RGB-D Images and Facial Landmarks for Potential Use in Computed-Aided Surgical Navigation of the Paranasal Sinus. In: Advances in Computer Graphics. Cham: Springer Nature Switzerland; 2024. pp. 135–145.

3. Mercan E, Atmosukarto I, Wu J, et al. Craniofacial Image Analysis. In: Health Monitoring and Personalized Feedback using Multimedia Data. Cham: Springer International Publishing; 2015. pp. 9–29.

4. Liu F, Zhao Q, Liu X, Zeng D. Joint Face Alignment and 3D Face Reconstruction with Application to Face Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2020; 42(3): 664-678. doi: 10.1109/tpami.2018.2885995

5. Sharma U, Faisal KN, Sharma RR, Arya KV. Facial Landmark-Based Human Emotion Recognition Technique for Oriented Viewpoints in the Presence of Facial Attributes. SN Computer Science. 2023; 4(3). doi: 10.1007/s42979-023-01727-y

6. Gupta A, Thakkar K, Gandhi V, Narayanan PJ. Nose, eyes and ears: Head pose estimation by locating facial keypoints. arXiv. 2018.

7. Cootes TF, Taylor CJ, Cooper DH, Graham J. Active Shape Models-Their Training and Application. Computer Vision and Image Understanding. 1995; 61(1): 38-59. doi: 10.1006/cviu.1995.1004

8. Cootes TF, Edwards GJ, Taylor CJ. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2001; 23(6): 681-685. doi: 10.1109/34.927467

9. Yang H, Zhu H, Wang Y, et al. FaceScape: A Large-Scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Published online June 2020. doi: 10.1109/cvpr42600.2020.00068

10. Farkas LG. Anthropometry of the head and face, 2nd ed. Available online: https://api.semanticscholar.org/CorpusID:57441646 (accessed on 23 March 2023).

11. Qiao H, Tan J, Wen S, et al. De Novo Dissecting the Three-Dimensional Facial Morphology of 2379 Han Chinese Individuals. Phenomics. Published online June 8, 2023. doi: 10.1007/s43657-023-00109-x

12. Liang Y, Liu H, Gao Z, et al. Ocular phenotype related SNP analysis in Southern Han Chinese population from Guangdong province. Gene. 2022; 826: 146458. doi: 10.1016/j.gene.2022.146458

13. Ercan I, Ozdemir ST, Etoz A, et al. Facial asymmetry in young healthy subjects evaluated by statistical shape analysis. Journal of Anatomy. 2008; 213(6): 663-669. doi: 10.1111/j.1469-7580.2008.01002.x

14. Sforza C, Rosati R, De Menezes M, et al. Three-Dimensional Computerized Anthropometry of the Nose. In: Handbook of Anthropometry. New York, NY: Springer New York; 2012. pp. 927–942.

15. Agbolade O, Nazri A, Yaakob R, et al. Morphometric approach to 3D soft-tissue craniofacial analysis and classification of ethnicity, sex, and age. Wilkinson C, ed. PLOS ONE. 2020; 15(4): e0228402. doi: 10.1371/journal.pone.0228402

16. Deutsch CK, Shell AR, Francis RW, Bird BD. The Farkas System of Craniofacial Anthropometry: Methodology and Normative Databases. In: Handbook of Anthropometry. New York, NY: Springer New York; 2012. pp. 561–573.

17. Zhuang W, Chen L, Hong C, et al. FT-GAN: Face Transformation with Key Points Alignment for Pose-Invariant Face Recognition. Electronics. 2019; 8(7): 807. doi: 10.3390/electronics8070807

18. Wang X, Wang Y, Li W, et al. Facial Expression Animation by Landmark Guided Residual Module. IEEE Transactions on Affective Computing. 2023; 14(2): 878-894. doi: 10.1109/taffc.2021.3100352

19. Otberdout N, Daoudi M, Kacem A, et al. Dynamic Facial Expression Generation on Hilbert Hypersphere With Conditional Wasserstein Generative Adversarial Nets. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2022; 44(2): 848-863. doi: 10.1109/tpami.2020.3002500

20. Kostinger M, Wohlhart P, Roth PM, Bischof H. Annotated Facial Landmarks in the Wild: A large-scale, real-world database for facial landmark localization. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain, Nov. 2011, pp. 2144–2151. doi: 10.1109/ICCVW.2011.6130513

21. Sagonas C, Antonakos E, Tzimiropoulos G, et al. 300 Faces In-The-Wild Challenge: database and results. Image and Vision Computing. 2016; 47: 3-18. doi: 10.1016/j.imavis.2016.01.002

22. Zhu X,Ramanan D. Face detection, pose estimation, and landmark localization in the wild. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, Jun. 2012, pp. 2879–2886. doi: 10.1109/CVPR.2012.6248014

23. Le V, Brandt J, Lin Z, et al. Interactive Facial Feature Localization. In Computer Vision—ECCV 2012. Berlin, Heidelberg: Springer Berlin Heidelberg; 2012. pp. 679–692. doi: 10.1007/978-3-642-33712-3_49

24. Deng J, Roussos A, Chrysos G, et al. The Menpo Benchmark for Multi-pose 2D and 3D Facial Landmark Localisation and Tracking. International Journal of Computer Vision. 2018; 127(6-7): 599-624. doi: 10.1007/s11263-018-1134-y

25. Zhu X, Lei Z, Liu X, et al. Face Alignment Across Large Poses: A 3D Solution. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Published online June 2016. doi: 10.1109/cvpr.2016.23

26. Wu Y, Ji Q. Facial Landmark Detection: A Literature Survey. International Journal of Computer Vision. 2018; 127(2): 115-142. doi: 10.1007/s11263-018-1097-z

27. Baker S, Matthews I. Lucas-Kanade 20 Years On: A Unifying Framework. Int. J. Comput. Vis. 2004; 56(3): 221–255. doi: 10.1023/B:VISI.0000011205.11775.fd

28. Matthews I, Baker S. Active Appearance Models Revisited. Int. J. Comput. Vis. 2004; 60(2): 135–164. doi: 10.1023/B:VISI.0000029666.37597.d3

29. Gross R, Matthews I, Baker S. Appearance-based face recognition and light-fields. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2004; 26(4): 449-465. doi: 10.1109/tpami.2004.1265861

30. Gross R, Matthews I, Baker S. Generic vs. person specific active appearance models. Image and Vision Computing. 2005; 23(12): 1080-1093. doi: 10.1016/j.imavis.2005.07.009

31. Cristinacce D, Cootes TF. Feature Detection and Tracking with Constrained Local Models. In: Proceedings of the British Machine Vision Conference 2006, Edinburgh, 2006, pp. 95.1-95.10. doi: 10.5244/C.20.95

32. Dantone M, Gall J, Fanelli G, Van Gool L. Real-time facial feature detection using conditional regression forests. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, Jun. 2012, pp. 2578–2585. doi: 10.1109/CVPR.2012.6247976

33. Zhang J, Hu H, Feng S. Robust Facial Landmark Detection via Heatmap-Offset Regression. IEEE Transactions on Image Processing. 2020; 29: 5050-5064. doi: 10.1109/tip.2020.2976765

34. Jeong M, Ko BC, Kwak S, Nam JY. Driver Facial Landmark Detection in Real Driving Situations. IEEE Transactions on Circuits and Systems for Video Technology. 2018; 28(10): 2753-2767. doi: 10.1109/tcsvt.2017.2769096

35. Deng Z, Liu H. Geometry-attentive relational reasoning for robust facial landmark detection. Neurocomputing. 2021; 453: 790-800. doi: 10.1016/j.neucom.2020.06.126

36. Wu Y, Ji Q. Discriminative Deep Face Shape Model for Facial Point Detection. International Journal of Computer Vision. 2014; 113(1): 37-53. doi: 10.1007/s11263-014-0775-8

37. Salazar A, Wuhrer S, Shu C, Prieto F. Fully automatic expression-invariant face correspondence. Machine Vision and Applications. 2013; 25(4): 859-879. doi: 10.1007/s00138-013-0579-9

38. Papazov C, Marks TK, Jones M. Real-time 3D head pose and facial landmark estimation from depth images using triangular surface patch features. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, Jun. 2015, pp. 4722–4730. doi: 10.1109/CVPR.2015.7299104

39. Berends B, Bielevelt F, Schreurs R, et al. Fully automated landmarking and facial segmentation on 3D photographs. arXiv. 2023.

40. Perakis P, Passalis G, Theoharis T, Kakadiaris IA. 3D Facial Landmark Detection under Large Yaw and Expression Variations. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2013; 35(7): 1552-1564. doi: 10.1109/tpami.2012.247

41. Wang J, Fang S, Fang M, et al. Automatic Landmark Placement for Large 3D Facial Image Dataset. In: 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA, Dec. 2019, pp. 5088–5093. doi: 10.1109/BigData47090.2019.9006310

42. Wang Z, Simoncelli EP, Bovik AC. Multiscale structural similarity for image quality assessment. In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, Pacific Grove, CA, USA, 2003, pp. 1398–1402. doi: 10.1109/ACSSC.2003.1292216

43. Zhou QY, Park J, Koltun V. Open3D: A Modern Library for 3D Data Processing. arXiv. 2018.

44. Ingale AK, Leema AA, Kim H, Udayan JD. Automatic 3D Facial Landmark-Based Deformation Transfer on Facial Variants for Blendshape Generation. Arabian Journal for Science and Engineering. 2022; 48(8): 10109-10123. doi: 10.1007/s13369-022-07403-2