Asia Pacific Academy of Science Pte. Ltd. (APACSCI) specializes in international journal publishing. APACSCI adopts the open access publishing model and provides an important communication bridge for academic groups whose interest fields include engineering, technology, medicine, computer, mathematics, agriculture and forestry, and environment.
Computer vision meets metaverse
Vol 5, Issue 1, 2024
Issue release: 30 June, 2024
VIEWS - 5742 (Abstract)
Download PDF
Abstract
This comprehensive analysis delves into the historical progression and important technological and contemporary advancements of computer vision inside the metaverse. The metaverse, which can be characterized as an interactive virtual reality environment that mirrors the physical world, signifies a novel domain for the utilization of computer vision in various applications. These applications span from object identification and tracking to gesture recognition and augmented reality. Additionally, a thorough evaluation of specific case studies occurs to provide a deeper understanding of the subject. Despite notable progress, the incorporation and utilization of computer vision inside the metaverse present numerous obstacles, including computational expenses, apprehensions regarding data privacy, and the faithful replication of physical aspects of reality. Potential solutions are examined, including deep learning approaches, optimization strategies, and the formation of ethical guidelines. A comprehensive analysis of anticipated patterns within the industry is also included, with emphasis on the confluence of artificial intelligence, the Internet of Things, and blockchain technologies. These convergences are predicted to create substantial prospects for the advancement of the metaverse. This review culminates by offering a contemplation on the ethical considerations and duties that arise from the utilization of computer vision in the realm beyond mortal existence. Research findings demonstrate that these technologies have greatly augmented user engagement and immersion within the digital domain. Readers can enhance their understanding of the interdependent connection between computer vision and the metaverse through the present analysis of existing scholarly works. Thus, this study aspires to make a valuable contribution to the advancement of research in this new domain.
Keywords
References
1. Lee LH, Braud T, Zhou P, et al. All one needs to know about metaverse: a complete survey on technological singularity, virtual ecosystem, and research agenda. Available online: https://arxiv.org/abs/2110.05352 (accessed on 2 June 2023).
2. Weinberger M. What is Metaverse?—A Definition Based on Qualitative Meta-Synthesis. Future Internet. 2022, 14(11): 310. doi: 10.3390/fi14110310
3. Li Z, Wu K, Jiang G, Yang Y. Extending depth of field by varifocal multi-view computational imaging for metaverse. Available online: https://ieeexplore.ieee.org/document/10337710/ (accessed on 2 June 2023).
4. Xi N, Chen J, Gama F, et al. The challenges of entering the metaverse: An experiment on the effect of extended reality on workload. Information Systems Frontiers. 2022, 25: 659-680. doi: 10.1007/s10796-022-10244-x
5. Kim DY, Lee HK, Chung K. Avatar-mediated experience in the metaverse: The impact of avatar realism on user-avatar relationship. Journal of Retailing and Consumer Services. 2023, 73: 103382. doi: 10.1016/j.jretconser.2023.103382
6. Zhang H, Lee S, Lu Y, et al. A Survey on Big Data Technologies and Their Applications to the metaverse: Past, Current and Future. Mathematics. 2022, 11(1): 96. doi: 10.3390/math11010096
7. Dwivedi YK, Hughes L, Baabdullah AM, et al. Metaverse beyond the hype: Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. International Journal of Information Management. 2022, 66: 102542. doi: 10.1016/j.ijinfomgt.2022.102542
8. Kitchenham B. Procedures for Performing Systematic Reviews. Keele University; 2004.
9. Nixon MS, Aguado AS. Feature Extraction and Image Processing for Computer Vision. Feature Extraction and Image Processing for Computer Vision. Elsevier; 2020.
10. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Communications of the ACM. 2017, 60(6): 84-90. doi: 10.1145/3065386
11. Stephenson N. Snow Crash. Spectra; 1992.
12. Tsou MH, Mejia C. Beyond mapping: extend the role of cartographers to user interface designers in the metaverse using virtual reality, augmented reality, and mixed reality. Cartography and Geographic Information Science. 2023, 1-15. doi: 10.1080/15230406.2023.2264748
13. The T, Pham Q, Pham X, et al. AI and computer vision technologies for metaverse. metaverse Communication and Computing Networks. In: Hoang DT, Nguyen DN, Nguyen CT, et al. (editors). Metaverse Communication and Computing Networks: Applications, Technologies, and Approaches. John Wiley & Sons; 2023. pp. 85-124. doi: 10.1002/9781394160013.ch5
14. Ray P, Bera A, Giri D, et al. Style matching CAPTCHA: match neural transferred styles to thwart intelligent attacks. Multimedia Systems. 2023, 29(4): 1865-1895. doi: 10.1007/s00530-023-01075-0
15. Braud T, Lee LH, Alhilal A, et al. DiOS—An Extended Reality Operating System for the metaverse. IEEE MultiMedia. 2023, 30(2): 70-80. doi: 10.1109/mmul.2022.3211351
16. Rai A, Harshit, Jain K, et al. Augmented reality in education and remote sensing. In: Proceedings of the IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium. 17–22 July 2022; Kuala Lumpur, Malaysia. pp. 6856-6859. doi: 10.1109/igarss46834.2022.9884101
17. Sin ZPT, Jia Y, Wu ACH, et al. Toward an Edu-metaverse of Knowledge: Immersive Exploration of University Courses. IEEE Transactions on Learning Technologies. 2023, 16(6): 1096-1110. doi: 10.1109/tlt.2023.3290814
18. Lin H, Wan S, Gan W, et al. Metaverse in education: vision, opportunities, and challenges. In: Proceedings of the 2022 IEEE International Conference on Big Data (Big Data). 17–20 December 2022; Osaka, Japan. pp. 2857-2866. doi: 10.1109/bigdata55660.2022.10021004
19. Jamshidi M (Behdad), Sargolzaei S, Foorginezhad S, et al. Metaverse and microorganism digital twins: A deep transfer learning approach. Applied Soft Computing. 2023, 147: 110798. doi: 10.1016/j.asoc.2023.110798
20. Juanes Méndez JA, Marcos-Pablos S, González Izard S. The metaverse in medical education and clinical practice. In: García-Peñalvo FJ, García-Holgado A (editors). In: Proceedings TEEM 2022: Tenth International Conference on Technological Ecosystems for Enhancing Multiculturality. Springer; 2023. pp. 157-164. doi: 10.1007/978-981-99-0942-1_15
21. Mitra S. metaverse: A Potential Virtual-Physical Ecosystem for Innovative Blended Education and Training. Journal of metaverse. 2023, 3(1): 66-72. doi: 10.57019/jmv.1168056
22. Sharma HK, Choudhury T. Hand gesture recognition. Available online: https://services.igi-global.com/resolvedoi/resolve.aspx?doi=10.4018/978-1-7998-9434-6.ch003 (accessed on 2 June 2023).
23. Tiwari SK, Girade PM, Khobe BP, et al. The novel approach of QR code detection with smart glasses. In: Proceedings of the 2022 10th International Conference on Emerging Trends in Engineering and Technology - Signal and Information Processing (ICETET-SIP-22); 29–30 April 2022; Nagpur, India. pp. 1-5. doi: 10.1109/icetet-sip-2254415.2022.9791569
24. Waisberg E, Ong J, Masalkhi M, et al. Meta smart glasses—Large language models and the future for assistive glasses for individuals with vision impairments. Eye. 2023. doi: 10.1038/s41433-023-02842-z
25. Jiang W, Li F, Mei L, et al. VisBLE: Vision-enhanced BLE device tracking. In: Proceedings of the 2022 19th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON); 20–23 September 2022; Stockholm, Sweden. pp. 217-225. doi: 10.1109/secon55815.2022.9918581
26. Yang J, Zhou Y, Huang H, et al. MetaFi: Device-free pose estimation via commodity WiFi for metaverse avatar simulation. In: Proceedings of the 2022 IEEE 8th World Forum on Internet of Things (WF-IoT); 26 October 2022–11 November 2022; Yokohama, Japan. doi: 10.1109/wf-iot54382.2022.10152057
27. Han Q, Zhao J, Lam KY. Facial landmark predictions with applications to metaverse. In: Proceedings of the 2022 IEEE 8th World Forum on Internet of Things (WF-IoT); 26 October 2022–11 November 2022; Yokohama, Japan. doi: 10.1109/wf-iot54382.2022.10152230
28. Song H, Yoon B, Cho W, et al. RC-SMPL: Real-time cumulative SMPL-based avatar body generation. In: Proceedings of the 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 16–20 October 2023; Sydney, Australia. pp. 88-98. doi: 10.1109/ismar59233.2023.00023
29. Abdelghafar S, Ezzat D, Darwish A, Hassanien AE. Metaverse for brain computer interface: towards new and improved applications. Available online: https://link.springer.com/10.1007/978-3-031-29132-6_3 (accessed on 2 June 2023).
30. Hamilton S. Deep Learning Computer Vision Algorithms, Customer Engagement Tools, and Virtual Marketplace Dynamics Data in the metaverse Economy. Journal of Self-Governance and Management Economics. 2022, 10(2): 37-51.
31. Szeliski R. Computer Vision: Algorithms and Applications. Springer Nat; 2022.
32. Tran NC, Wang J, Vu TH, et al. Anti-aliasing convolution neural network of finger vein recognition for virtual reality (VR) human–robot equipment of metaverse. The Journal of Supercomputing. 2022, 79(3): 2767-2782. doi: 10.1007/s11227-022-04680-4
33. Rawat W, Wang Z. Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review. Neural Computation. 2017, 29(9): 2352-449.
34. Redmon J, Divvala S, Girshick R, et al. You only look once: Unified, real-time object detection. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 27–30 June 2016; Las Vegas, NV, USA. pp. 779-788. doi: 10.1109/cvpr.2016.91
35. Garcia-Garcia A, Orts-Escolano S, Oprea S, et al. A survey on deep learning techniques for image and video semantic segmentation. Applied Soft Computing. 2018, 70: 41-65. doi: 10.1016/j.asoc.2018.05.018
36. Nilsson J, Odblom ACE, Fredriksson J, et al. Performance evaluation method for mobile computer vision systems using augmented reality. In: Proceedings of the 2010 IEEE Virtual Reality Conference (VR); 20–24 March 2010; Boston, MA, USA. pp. 19-22. doi: 10.1109/vr.2010.5444821
37. Zhang X, Min G, Li T, Ma Z, Cao X, Wang S. AI and Blockchain Empowered metaverse for Web 3.0: Vision, Architecture, and Future Directions. IEEE Communications Magazine. 2023, 61(8): 60-6.
38. Dubey A, Bhardwaj N, Upadhyay A, Ramnani R. AI for immersive metaverse experience. In: Proceedings of the 6th Joint International Conference on Data Science & Management of Data; 4–7 January 2023; Mumbai, India. pp. 316-319. doi: 10.1145/3570991.3571045
39. Saffo D, Di Bartolomeo S, Yildirim C, Dunne C. Remote and Collaborative Virtual Reality Experiments via Social VR Platforms. In: Proceedings of the CHI’ 21: CHI Conference on Human Factors in Computing Systems; 8–13 May 2021; Yokohama, Japan. pp. 1-15. doi: 10.1145/3411764.3445426
40. Sun Y, Xu Y, Cheng C, et al. Travel with Wander in the metaverse: An AI chatbot to Visit the Future Earth. In: Proceedings of the 2022 IEEE 24th International Workshop on Multimedia Signal Processing (MMSP); 26–28 September 2022; Shanghai, China. doi: 10.1109/MMSP55362.2022.9950031
41. Jovanović A, Milosavljević A. VoRtex metaverse Platform for Gamified Collaborative Learning. Electronics. 2022, 11(3): 317. doi: 10.3390/electronics11030317
42. Tayal S, Rajagopal K, Mahajan V. Virtual Reality based metaverse of Gamification. In: Proceedings of the 2022 6th International Conference on Computing Methodologies and Communication (ICCMC); 29–31 March 2022; Erode, India. pp. 1597-1604. doi: 10.1109/iccmc53470.2022.9753727
43. Mystakidis S. metaverse. Encyclopedia. 2022, 2(1): 486-497. doi: 10.3390/encyclopedia2010031
44. Sun Z, Zhu M, Shan X, et al. Augmented tactile-perception and haptic-feedback rings as human-machine interfaces aiming for immersive interactions. Nature Communications. 2022, 13(1). doi: 10.1038/s41467-022-32745-8
45. Cao L. Decentralized AI: Edge Intelligence and Smart Blockchain, metaverse, Web3, and DeSci. IEEE Intelligent Systems. 2022, 37(3): 6-19. doi: 10.1109/mis.2022.3181504
46. Lim WYB, Xiong Z, Niyato D, et al. Realizing the metaverse with Edge Intelligence: A Match Made in Heaven. IEEE Wireless Communications. 2023, 30(4): 64-71. doi: 10.1109/mwc.018.2100716
47. Picone M, Mariani S, Virdis A, et al. Digital Twin & Blockchain: Technology Enablers for metaverse Computing. In: Proceedings of the 2023 IEEE International Conference on metaverse Computing, Networking and Applications (MetaCom). 26–28 June 2023; Kyoto, Japan. doi: 10.1109/metacom57706.2023.00017
48. Chen B, Song C, Lin B, et al. A Cross-platform metaverse Data Management System. In: Proceedings of the 2022 IEEE International Conference on Metrology for Extended Reality, Artificial Intelligence and Neural Engineering (MetroXRAINE); 26–28 October 2022; Rome, Italy. pp. 145-150. doi: 10.1109/metroxraine54828.2022.9967588
49. Lo SC, Tsai HH. Design of 3D Virtual Reality in the metaverse for Environmental Conservation Education Based on Cognitive Theory. Sensors. 2022, 22(21): 8329. doi: 10.3390/s22218329
50. Wang Y, Siau KL, Wang L. metaverse and human-computer interaction: A technology framework for 3D virtual worlds. In: Chen JYC, Fragomeni G, Degen H, Ntoa S (editors). HCI International 2022 – Late Breaking Papers: Interacting with eXtended Reality and Artificial Intelligence, Proceedings of the 24th International Conference on Human-Computer Interaction; 26 June 2022–1 July 2022; Online Conference. Springer; 2022. Volume 13518. pp. 213-221. Springer, Cham. https://doi.org/10.1007/978-3-031-21707-4_16
51. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015, 521(7553): 436-444. doi: 10.1038/nature14539
52. Lee HM, Ham SM, Moon H, et al. A metaverse Emotion Mapping System with an AIoT Facial Expression Recognition Device. In: Proceedings of the 2023 IEEE International Conference on metaverse Computing, Networking and Applications (MetaCom). 26–28 June 2023; Kyoto, Japan. pp. 704-707. doi: 10.1109/metacom57706.2023.00132
53. He S, Zhao H, Yu L. The avatar facial expression reenactment method in the metaverse based on overall-local optical-flow estimation and illumination difference. In: Proceedings of the 2023 26th International Conference on Computer Supported Cooperative Work in Design (CSCWD); 24–26 May 2023; Rio de Janeiro, Brazil. pp. 1312-1317. doi: 10.1109/cscwd57460.2023.10152763
54. Jang JY. Analyzing visual behavior of consumers in a virtual reality fashion store using eye tracking. Fashion and Textiles. 2023, 10(1). doi: 10.1186/s40691-023-00345-9
55. Hendrikx M, Meijer S, Van Der Velden J, et al. Procedural content generation for games. ACM Transactions on Multimedia Computing, Communications, and Applications. 2013, 9(1): 1-22. doi: 10.1145/2422956.2422957
56. Kipper G, Rampolla J. Augmented Reality. In: Augmented Reality: An Emerging Technologies Guide to AR, 1st ed. Elsevier; 2013.
57. Furukawa Y, Hernández C. Multi-View Stereo: A Tutorial. Foundations and Trends® in Computer Graphics and Vision. 2015, 9(1-2): 1-148. doi: 10.1561/0600000052
58. Elhagry A. Text-to-metaverse: Towards a Digital Twin-Enabled Multimodal Conditional Generative metaverse. In: Proceedings of the 31st ACM International Conference on Multimedia; 29 October 2023–3 November 2023; Ottawa, ON, Canada. pp. 9336-9339. doi: 10.1145/3581783.3613432
59. Wang X, Chen Q, Li Z. A 3D Reconstruction Method for Augmented Reality Sandbox Based on Depth Sensor. In: Proceedings of the 2021 IEEE 2nd International Conference on Information Technology, Big Data and Artificial Intelligence (ICIBA). 17–19 December 2021; Chongqing, China. pp. 844-849. doi: 10.1109/iciba52610.2021.9687867
60. Wang Y, Su Z, Zhang N, Liu D, et al. A survey on metaverse: Fundamentals, security, and privacy. Available online: https://arxiv.org/abs/2203.02662 (accessed on 2 June 2023).
61. Cheng R, Wu N, Varvello M, et al. Are we ready for metaverse? In: Proceedings of the 22nd ACM Internet Measurement Conference. 25–27 October 2022; New York, United States. pp. 504-518. doi: 10.1145/3517745.3561417
62. Isaac M. Facebook’s new bet on virtual reality: Conference rooms. Available online: https://go.gale.com/ps/i.do?id=GALE%7CA672687966&sid=googleScholar&v=2.1&it=r&linkaccess=abs&issn=22699740&p=AONE&sw=w&userGroupName=anon%7Efcd9bb5b&aty=open-web-entry (accessed on 6 February 2024).
63. Meta. Introducing horizon workrooms: remote collaboration reimagined. Available online: https://www.meta.com/blog/quest/workrooms/ (accessed on 6 February 2024).
64. Shiau WL, Huang LC. Scale development for analyzing the fit of real and virtual world integration: an example of Pokémon Go. Information Technology & People. 2022, 36(2): 500-531. doi: 10.1108/itp-11-2020-0793
65. Niantic Inc. Pockemon go. Available online: https://pokemongolive.com/ (accessed on 2 February 2024).
66. Bonetti F, Perry P. A Review of Consumer-Facing Digital Technologies Across Different Types of Fashion Store Formats. IGI Global; 2017.
67. FashionNetwork. Asos invests in AR technology with new feature. Available online: https://uk.fashionnetwork.com/news/Asos-invests-in-ar-technology-with-new-feature,1109510.html (accessed on 2 June 2023).
68. Egliston B, Carter M. Critical questions for Facebook’s virtual reality: data, power and the metaverse. Internet Policy Review. 2021, 10(4). doi: 10.14763/2021.4.1610
69. Buolamwini J, Gebru T. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Machine Learning Research; 2018.
70. Xu M, Ng WC, Lim WYB, et al. A Full Dive into Realizing the Edge-Enabled metaverse: Visions, Enabling Technologies, and Challenges. IEEE Communications Surveys & Tutorials. 2023, 25(1): 656-700. doi: 10.1109/comst.2022.3221119
71. Shen S, Zhang W. A method for synthesizing dynamic image of virtual human. In: Proceedings of the 2023 3rd International Conference on Consumer Electronics and Computer Engineering (ICCECE). 6–8 January 2023; Guangzhou, China. pp. 698-702. doi: 10.1109/iccece58074.2023.10135229
72. Chang YT. Kernel-wise difference minimization for convolutional neural network compression in metaverse. Available online: https://www.frontiersin.org/articles/10.3389/fdata.2023.1200382/full (accessed on 2 June 2023).
73. Boyer S, Veeramachaneni K. Transfer learning for predictive models in massive open online courses. In: Conati C, Heffernan N, Mitrovic A, Verdejo M (editors). Artificial Intelligence in Education, Proceedings of the 17th International Conference, AIED 2015; 22–26 June 2016; Madrid, Spain. Springer; 2015. Volume 9112. pp. 54-63. doi: 10.1007/978-3-319-19773-9_6
74. Yaqob M, Hafez MM. Metaverse—An overview of daily usage and risks. In: Proceedings of the 2022 OPJU International Technology Conference on Emerging Technologies for Sustainable Development (OTCON); 8–10 February 2023; Raigarh, Chhattisgarh, India. doi: 10.1109/otcon56053.2023.10113922
75. Yosinski J, Clune J, Bengio Y, Lipson H. How transferable are features in deep neural networks? In: Ghahramani Z, Welling M, Cortes C, et al. (editors). Advances in Neural Information Processing Systems 27. Neural Information Processing Systems; 2014.
76. Al-Ghaili AM, Kasim H, Al-Hada NM, et al. A Review of metaverse’s Definitions, Architecture, Applications, Challenges, Issues, Solutions, and Future Trends. IEEE Access. 2022, 10: 125835-125866. doi: 10.1109/access.2022.3225638
77. Zhang L, Qiu Q, Lin H, et al. DreamFace: Progressive Generation of Animatable 3D Faces under Text Guidance. ACM Transactions on Graphics. 2023, 42(4): 1-16. doi: 10.1145/3592094
78. Baltrusaitis T, Ahuja C, Morency LP. Multimodal Machine Learning: A Survey and Taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2019, 41(2): 423-443. doi: 10.1109/tpami.2018.2798607
79. Smart P. Minds in the Metaverse: Extended Cognition Meets Mixed Reality. Philosophy & Technology. 2022, 35(4). doi: 10.1007/s13347-022-00580-w
80. Xu J, Papangelis K, Dunham J, et al. metaverse: The Vision for the Future. In: Proceedings of the CHI’22: CHI Conference on Human Factors in Computing Systems; 29 April 2022; 5 May 2022; New Orleans, LA, USA. pp. 1-3. doi: 10.1145/3491101.3516399
81. Wang P, Wei L, Sun W, et al. Energy-Efficient Distributed Learning and Sharding Blockchain for Sustainable Metaverse. IEEE Wireless Communications. 2023, 30(5): 128-134. doi: 10.1109/mwc.015.2300107
82. Haeberlen A, Phan LTX, McGuire M. metaverse as a Service: Megascale Social 3D on the Cloud. In: Proceedings of the 2023 ACM Symposium on Cloud Computing; 30 October 2023–1 November 2023; Santa Cruz, CA, USA. pp. 298-307. doi: 10.1145/3620678.3624662
83. Brownsword R, Scotford E, Yeung K (editors). Regulating in the Face of Sociotechnical Change. Oxford University Press; 2016. doi: 10.1093/oxfordhb/9780199680832.013.49
84. Riccio P, Oliver N. Racial bias in the beautyverse: Evaluation of augmented-reality beauty filters. Available online: https://link.springer.com/10.1007/978-3-031-25066-8_43 (accessed on 2 June 2023).
85. Belk R, Humayun M, Brouard M. Money, possessions, and ownership in the metaverse: NFTs, cryptocurrencies, Web3 and Wild Markets. Journal of Business Research. 2022, 153: 198-205. doi: 10.1016/j.jbusres.2022.08.031
Refbacks
- There are currently no refbacks.
Copyright (c) 2024 Vasiliki Zakynthinou, Venetis Kanakaris, Eleni Vrochidou, George A. Papakostas
This work is licensed under a Creative Commons Attribution 4.0 International License.
This site is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).
Prof. Zhigeng Pan
Director, Institute for Metaverse, Nanjing University of Information Science & Technology, China
Prof. Jianrong Tan
Academician, Chinese Academy of Engineering, China
Processing Speed (2023)
-
-
- ＜7 days: submission to screening review decision
- 36 days: received to accepted
- 56 days: received to online
-
