Markerless Facial Reconstruction Motion Capture Using Triangulation Method

Muhammad Alwali, Sevito Fernanda Pambudi, Laras Suciningtyas, Eko Mulyanto Yuniarno

Abstract


Motion capture is a popular research topic, with one of its main applications being human face reconstruction. The demand for converting 2D images into 3D reconstructions continues to increase, especially in facial reconstruction, where progress is made in improving the accuracy of facial position prediction. However, there is still a significant gap in developing facial reconstruction technologies that can consistently convert 2D to 3D data with high accuracy, especially in scenarios involving dynamic facial expressions, diverse facial angles, and complex environmental conditions. Therefore, an approach using the triangulation method for 3D face reconstruction in the real world was developed. In the experiments, two cameras were used to obtain two face landmark coordinates so that the triangulation method can be implemented for 3D face reconstruction. This research aims to develop a motion capture approach that is able to accurately and efficiently transform 2D data into 3D face models without the need for complex hardware. The main contribution of this research is the development of a machine learning-based markerless motion capture technique designed to improve the accuracy of face position prediction in 3D face reconstruction from 2D data in realistic environments. This method seeks to bridge the current technology gap by providing a more flexible and reliable solution, expanding the potential applications of motion capture in various fields without dependence on specialized hardware. The results of face reconstruction research using markerless motion capture and triangulation method show RMSE values of 3.560839 for eyes, 1.644749 for nose, and 4.054638 for lips.


Full Text:

PDF

References


Q. Zhuang, Z. Kehua, J. Wang, and Q. Chen, “Driver fatigue detection method based on eye states with pupil and iris segmentation,” IEEE Access, vol. 8, 2020, doi: 10.1109/ACCESS.2020.3025818.

H. O. Shahreza and S. Marcel, “Comprehensive Vulnerability Evaluation of Face Recognition Systems to Template Inversion Attacks via 3D Face Reconstruction,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 12, 2023, doi: 10.1109/TPAMI.2023.3312123.

F. Schroff, D. Kalenichenko, and J. Philbin, “FaceNet: A unified embedding for face recognition and clustering,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2015, vol. 07-12-June-2015, doi: 10.1109/CVPR.2015.7298682.

U. Solichah, M. H. Purnomo, and E. M. Yuniarno, “Marker-less Motion Capture Based on Openpose Model Using Triangulation,” 2020, doi: 10.1109/ISITIA49792.2020.9163662.

X. Zhu, C. Yu, D. Huang, Z. Lei, H. Wang, and S. Z. Li, “Beyond 3DMM: Learning to Capture High-Fidelity 3D Face Shape,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 2, 2023, doi: 10.1109/TPAMI.2022.3164131.

G. Tao, S. Sun, S. Huang, Z. Huang, and J. Wu, “Human modeling and real-time motion reconstruction for micro-sensor motion capture,” 2011, doi: 10.1109/VECIMS.2011.6052193.

J. Jo, Y. J. Jung, and J. Kim, “3D face reconstruction from one side-view face images,” 2014, doi: 10.1109/ELINFOCOM.2014.6914359.

S. Bharadwaj, T. I. Dhamecha, M. Vatsa, and R. Singh, “Computationally efficient face spoofing detection with motion magnification,” 2013, doi: 10.1109/CVPRW.2013.23.

Y. W. Cha et al., “Towards fully mobile 3D face, body, and environment capture using only head-worn cameras,” IEEE Trans. Vis. Comput. Graph., vol. 24, no. 11, 2018, doi: 10.1109/TVCG.2018.2868527.

B. Hu et al., “Face Landmark Calibration Based on 3D Reconstruction and Deep Learning,” 2022, doi: 10.1109/DDCLS55054.2022.9858436.

W. Peng et al., “IE-aware Consistency Losses for Detailed 3D Face Reconstruction from Multiple Images in the Wild,” in 2024 IEEE International Conference on Multimedia and Expo (ICME), 2024, pp. 1–6, doi: 10.1109/ICME57554.2024.10687861.

C. Darujati and M. Hariadi, “Facial motion cpture with 3D active appearance models,” 2013, doi: 10.1109/ICICI-BME.2013.6698465.

Y. Zhang, X. He, Y. Hu, J. Zeng, H. Yang, and S. Zhou, “Face animation making method based on facial motion capture,” 2021, doi: 10.1109/ICESIT53460.2021.9696547.

L. Shi, Z. An, J. Zhao, L. Wang, and Q. Du, “A study of face motion capture and its data processing technique applied to the speech training of hearing-impaired children,” 2012, doi: 10.1109/ICINIS.2012.63.

A. N. Ansari, M. Abdel-Mottaleb, and M. H. Mahoor, “3D face mesh modeling from range images for 3D face recognition,” in Proceedings - International Conference on Image Processing, ICIP, 2007, vol. 4, doi: 10.1109/ICIP.2007.4380066.

G. Wen, Y. Yang, Z. Yang, J. Yue, and S. Yan, “A High-accuracy Camera Calibration Method Based on Special Circular Target,” in 2022 International Conference on Industrial Automation, Robotics and Control Engineering (IARCE), 2022, pp. 24–28, doi: 10.1109/IARCE57187.2022.00015.

S. Bi, Y. Gu, Z. Zhang, H. Liu, C. Zhai, and M. Gong, “Multi-camera stereo vision based on weights,” 2020, doi: 10.1109/I2MTC43012.2020.9128927.

C. Y. Vincent and T. Tjahjadi, “Multiview camera-calibration framework for nonparametric distortions removal,” IEEE Trans. Robot., vol. 21, no. 5, 2005, doi: 10.1109/TRO.2005.851383.

D. M. and A. T. P. S. Bisht, P. Colantoni, “MultiView Markerless MoCap - MultiView Performance Capture, 3D Pose Motion Reconstruction and Comparison.” pp. 333–340, 2023, doi: 10.1109/SITIS61268.2023.00061.

M.Meghana, M.Vasavi, and D. D. Shravani, “FACIAL LANDMARK DETECTION WITH MEDIAPIPE & CREATING ANIMATED SNAPCHAT FILTERS,” Int. J. Innov. Eng. Manag. Res., 2022, doi: 10.48047/ijiemr/v11/i06/10.

A. M. Al-Nuimi and G. J. Mohammed, “Face Direction Estimation based on Mediapipe Landmarks,” 2021, doi: 10.1109/ICCITM53167.2021.9677878.

N. Kumar Rao B, N. Panini Challa, E. S. P. Krishna, and S. S. Chakravarthi, “Facial Landmarks Detection System with OpenCV Mediapipe and Python using Optical Flow (Active) Approach,” 2023, doi: 10.1109/ICACITE57410.2023.10182585.

C. Zhang and Z. Zhang, “Improving multiview face detection with multi-task deep convolutional neural networks,” 2014, doi: 10.1109/WACV.2014.6835990.

S. V. F. Barreto, R. E. Sant’Anna, and M. A. F. Feitosa, “A method for image processing and distance measuring based on laser distance triangulation,” 2013, doi: 10.1109/ICECS.2013.6815509.

J. Lin, M. Zhao, G. Yin, H. Zhou, T. Hudoyberdi, and B. Jiang, “A Method for Depth Camera Calibration Based on Motion Capture System,” 2023, doi: 10.1109/ICCD59681.2023.10420758.

Z.-H. Feng, P. Huber, J. Kittler, P. J. B. Hancock, X.-J. Wu, Q. Zhao, P. Koppen, and M. Rätsch, "Evaluation of Dense 3D Reconstruction from 2D Face Images in the Wild," arXiv preprint arXiv:1803.05536, 2018. [Online]. Available: https://arxiv.org/abs/1803.05536.

W. Zhao, C. Yang, J. Ye, R. Zhang, Y. Yan, X. Yang, B. Dong, A. Hussain, and K. Huang, "From 2D Images to 3D Model: Weakly Supervised Multi-View Face Reconstruction with Deep Fusion," arXiv preprint arXiv:2204.03842, 2024. [Online]. Available: https://arxiv.org/abs/2204.03842.




DOI: https://doi.org/10.12962/jaree.v9i2.456

Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.