Open Journal Systems

Computer vision meets metaverse

Vasiliki Zakynthinou, Venetis Kanakaris, Eleni Vrochidou, George A. Papakostas

Article ID: 2464
Vol 5, Issue 1, 2024

DOI: https://doi.org/10.54517/m.v5i1.2464
VIEWS - 229 (Abstract)

Abstract

This comprehensive analysis delves into the historical progression and important technological and contemporary advancements of computer vision inside the metaverse. The metaverse, which can be characterized as an interactive virtual reality environment that mirrors the physical world, signifies a novel domain for the utilization of computer vision in various applications. These applications span from object identification and tracking to gesture recognition and augmented reality. Additionally, a thorough evaluation of specific case studies occurs to provide a deeper understanding of the subject. Despite notable progress, the incorporation and utilization of computer vision inside the metaverse present numerous obstacles, including computational expenses, apprehensions regarding data privacy, and the faithful replication of physical aspects of reality. Potential solutions are examined, including deep learning approaches, optimization strategies, and the formation of ethical guidelines. A comprehensive analysis of anticipated patterns within the industry is also included, with emphasis on the confluence of artificial intelligence, the Internet of Things, and blockchain technologies. These convergences are predicted to create substantial prospects for the advancement of the metaverse. This review culminates by offering a contemplation on the ethical considerations and duties that arise from the utilization of computer vision in the realm beyond mortal existence. Research findings demonstrate that these technologies have greatly augmented user engagement and immersion within the digital domain. Readers can enhance their understanding of the interdependent connection between computer vision and the metaverse through the present analysis of existing scholarly works. Thus, this study aspires to make a valuable contribution to the advancement of research in this new domain.

Keywords

computer vision; metaverse; visual world; IoT; blockchain; artificial intelligence

Full Text:

PDF

References

1. Lee LH, Braud T, Zhou P, et al. All one needs to know about metaverse: a complete survey on technological singularity, virtual ecosystem, and research agenda. Available online: https://arxiv.org/abs/2110.05352 (accessed on 2 June 2023).

2. Weinberger M. What is Metaverse?—A Definition Based on Qualitative Meta-Synthesis. Future Internet. 2022, 14(11): 310. doi: 10.3390/fi14110310

3. Li Z, Wu K, Jiang G, Yang Y. Extending depth of field by varifocal multi-view computational imaging for metaverse. Available online: https://ieeexplore.ieee.org/document/10337710/ (accessed on 2 June 2023).

4. Xi N, Chen J, Gama F, et al. The challenges of entering the metaverse: An experiment on the effect of extended reality on workload. Information Systems Frontiers. 2022, 25: 659-680. doi: 10.1007/s10796-022-10244-x

5. Kim DY, Lee HK, Chung K. Avatar-mediated experience in the metaverse: The impact of avatar realism on user-avatar relationship. Journal of Retailing and Consumer Services. 2023, 73: 103382. doi: 10.1016/j.jretconser.2023.103382

6. Zhang H, Lee S, Lu Y, et al. A Survey on Big Data Technologies and Their Applications to the metaverse: Past, Current and Future. Mathematics. 2022, 11(1): 96. doi: 10.3390/math11010096

7. Dwivedi YK, Hughes L, Baabdullah AM, et al. Metaverse beyond the hype: Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. International Journal of Information Management. 2022, 66: 102542. doi: 10.1016/j.ijinfomgt.2022.102542

8. Kitchenham B. Procedures for Performing Systematic Reviews. Keele University; 2004.

9. Nixon MS, Aguado AS. Feature Extraction and Image Processing for Computer Vision. Feature Extraction and Image Processing for Computer Vision. Elsevier; 2020.

10. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Communications of the ACM. 2017, 60(6): 84-90. doi: 10.1145/3065386

11. Stephenson N. Snow Crash. Spectra; 1992.

12. Tsou MH, Mejia C. Beyond mapping: extend the role of cartographers to user interface designers in the metaverse using virtual reality, augmented reality, and mixed reality. Cartography and Geographic Information Science. 2023, 1-15. doi: 10.1080/15230406.2023.2264748

13. The T, Pham Q, Pham X, et al. AI and computer vision technologies for metaverse. metaverse Communication and Computing Networks. In: Hoang DT, Nguyen DN, Nguyen CT, et al. (editors). Metaverse Communication and Computing Networks: Applications, Technologies, and Approaches. John Wiley & Sons; 2023. pp. 85-124. doi: 10.1002/9781394160013.ch5

14. Ray P, Bera A, Giri D, et al. Style matching CAPTCHA: match neural transferred styles to thwart intelligent attacks. Multimedia Systems. 2023, 29(4): 1865-1895. doi: 10.1007/s00530-023-01075-0

15. Braud T, Lee LH, Alhilal A, et al. DiOS—An Extended Reality Operating System for the metaverse. IEEE MultiMedia. 2023, 30(2): 70-80. doi: 10.1109/mmul.2022.3211351

16. Rai A, Harshit, Jain K, et al. Augmented reality in education and remote sensing. In: Proceedings of the IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium. 17–22 July 2022; Kuala Lumpur, Malaysia. pp. 6856-6859. doi: 10.1109/igarss46834.2022.9884101

17. Sin ZPT, Jia Y, Wu ACH, et al. Toward an Edu-metaverse of Knowledge: Immersive Exploration of University Courses. IEEE Transactions on Learning Technologies. 2023, 16(6): 1096-1110. doi: 10.1109/tlt.2023.3290814

18. Lin H, Wan S, Gan W, et al. Metaverse in education: vision, opportunities, and challenges. In: Proceedings of the 2022 IEEE International Conference on Big Data (Big Data). 17–20 December 2022; Osaka, Japan. pp. 2857-2866. doi: 10.1109/bigdata55660.2022.10021004

19. Jamshidi M (Behdad), Sargolzaei S, Foorginezhad S, et al. Metaverse and microorganism digital twins: A deep transfer learning approach. Applied Soft Computing. 2023, 147: 110798. doi: 10.1016/j.asoc.2023.110798

20. Juanes Méndez JA, Marcos-Pablos S, González Izard S. The metaverse in medical education and clinical practice. In: García-Peñalvo FJ, García-Holgado A (editors). In: Proceedings TEEM 2022: Tenth International Conference on Technological Ecosystems for Enhancing Multiculturality. Springer; 2023. pp. 157-164. doi: 10.1007/978-981-99-0942-1_15

21. Mitra S. metaverse: A Potential Virtual-Physical Ecosystem for Innovative Blended Education and Training. Journal of metaverse. 2023, 3(1): 66-72. doi: 10.57019/jmv.1168056

22. Sharma HK, Choudhury T. Hand gesture recognition. Available online: https://services.igi-global.com/resolvedoi/resolve.aspx?doi=10.4018/978-1-7998-9434-6.ch003 (accessed on 2 June 2023).

23. Tiwari SK, Girade PM, Khobe BP, et al. The novel approach of QR code detection with smart glasses. In: Proceedings of the 2022 10th International Conference on Emerging Trends in Engineering and Technology - Signal and Information Processing (ICETET-SIP-22); 29–30 April 2022; Nagpur, India. pp. 1-5. doi: 10.1109/icetet-sip-2254415.2022.9791569

24. Waisberg E, Ong J, Masalkhi M, et al. Meta smart glasses—Large language models and the future for assistive glasses for individuals with vision impairments. Eye. 2023. doi: 10.1038/s41433-023-02842-z

25. Jiang W, Li F, Mei L, et al. VisBLE: Vision-enhanced BLE device tracking. In: Proceedings of the 2022 19th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON); 20–23 September 2022; Stockholm, Sweden. pp. 217-225. doi: 10.1109/secon55815.2022.9918581

26. Yang J, Zhou Y, Huang H, et al. MetaFi: Device-free pose estimation via commodity WiFi for metaverse avatar simulation. In: Proceedings of the 2022 IEEE 8th World Forum on Internet of Things (WF-IoT); 26 October 2022–11 November 2022; Yokohama, Japan. doi: 10.1109/wf-iot54382.2022.10152057

27. Han Q, Zhao J, Lam KY. Facial landmark predictions with applications to metaverse. In: Proceedings of the 2022 IEEE 8th World Forum on Internet of Things (WF-IoT); 26 October 2022–11 November 2022; Yokohama, Japan. doi: 10.1109/wf-iot54382.2022.10152230

28. Song H, Yoon B, Cho W, et al. RC-SMPL: Real-time cumulative SMPL-based avatar body generation. In: Proceedings of the 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 16–20 October 2023; Sydney, Australia. pp. 88-98. doi: 10.1109/ismar59233.2023.00023

29. Abdelghafar S, Ezzat D, Darwish A, Hassanien AE. Metaverse for brain computer interface: towards new and improved applications. Available online: https://link.springer.com/10.1007/978-3-031-29132-6_3 (accessed on 2 June 2023).

30. Hamilton S. Deep Learning Computer Vision Algorithms, Customer Engagement Tools, and Virtual Marketplace Dynamics Data in the metaverse Economy. Journal of Self-Governance and Management Economics. 2022, 10(2): 37-51.

31. Szeliski R. Computer Vision: Algorithms and Applications. Springer Nat; 2022.

32. Tran NC, Wang J, Vu TH, et al. Anti-aliasing convolution neural network of finger vein recognition for virtual reality (VR) human–robot equipment of metaverse. The Journal of Supercomputing. 2022, 79(3): 2767-2782. doi: 10.1007/s11227-022-04680-4

33. Rawat W, Wang Z. Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review. Neural Computation. 2017, 29(9): 2352-449.

34. Redmon J, Divvala S, Girshick R, et al. You only look once: Unified, real-time object detection. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 27–30 June 2016; Las Vegas, NV, USA. pp. 779-788. doi: 10.1109/cvpr.2016.91

35. Garcia-Garcia A, Orts-Escolano S, Oprea S, et al. A survey on deep learning techniques for image and video semantic segmentation. Applied Soft Computing. 2018, 70: 41-65. doi: 10.1016/j.asoc.2018.05.018

36. Nilsson J, Odblom ACE, Fredriksson J, et al. Performance evaluation method for mobile computer vision systems using augmented reality. In: Proceedings of the 2010 IEEE Virtual Reality Conference (VR); 20–24 March 2010; Boston, MA, USA. pp. 19-22. doi: 10.1109/vr.2010.5444821

37. Zhang X, Min G, Li T, Ma Z, Cao X, Wang S. AI and Blockchain Empowered metaverse for Web 3.0: Vision, Architecture, and Future Directions. IEEE Communications Magazine. 2023, 61(8): 60-6.

38. Dubey A, Bhardwaj N, Upadhyay A, Ramnani R. AI for immersive metaverse experience. In: Proceedings of the 6th Joint International Conference on Data Science & Management of Data; 4–7 January 2023; Mumbai, India. pp. 316-319. doi: 10.1145/3570991.3571045

39. Saffo D, Di Bartolomeo S, Yildirim C, Dunne C. Remote and Collaborative Virtual Reality Experiments via Social VR Platforms. In: Proceedings of the CHI’ 21: CHI Conference on Human Factors in Computing Systems; 8–13 May 2021; Yokohama, Japan. pp. 1-15. doi: 10.1145/3411764.3445426

40. Sun Y, Xu Y, Cheng C, et al. Travel with Wander in the metaverse: An AI chatbot to Visit the Future Earth. In: Proceedings of the 2022 IEEE 24th International Workshop on Multimedia Signal Processing (MMSP); 26–28 September 2022; Shanghai, China. doi: 10.1109/MMSP55362.2022.9950031

41. Jovanović A, Milosavljević A. VoRtex metaverse Platform for Gamified Collaborative Learning. Electronics. 2022, 11(3): 317. doi: 10.3390/electronics11030317

42. Tayal S, Rajagopal K, Mahajan V. Virtual Reality based metaverse of Gamification. In: Proceedings of the 2022 6th International Conference on Computing Methodologies and Communication (ICCMC); 29–31 March 2022; Erode, India. pp. 1597-1604. doi: 10.1109/iccmc53470.2022.9753727

43. Mystakidis S. metaverse. Encyclopedia. 2022, 2(1): 486-497. doi: 10.3390/encyclopedia2010031

44. Sun Z, Zhu M, Shan X, et al. Augmented tactile-perception and haptic-feedback rings as human-machine interfaces aiming for immersive interactions. Nature Communications. 2022, 13(1). doi: 10.1038/s41467-022-32745-8

45. Cao L. Decentralized AI: Edge Intelligence and Smart Blockchain, metaverse, Web3, and DeSci. IEEE Intelligent Systems. 2022, 37(3): 6-19. doi: 10.1109/mis.2022.3181504

46. Lim WYB, Xiong Z, Niyato D, et al. Realizing the metaverse with Edge Intelligence: A Match Made in Heaven. IEEE Wireless Communications. 2023, 30(4): 64-71. doi: 10.1109/mwc.018.2100716

47. Picone M, Mariani S, Virdis A, et al. Digital Twin & Blockchain: Technology Enablers for metaverse Computing. In: Proceedings of the 2023 IEEE International Conference on metaverse Computing, Networking and Applications (MetaCom). 26–28 June 2023; Kyoto, Japan. doi: 10.1109/metacom57706.2023.00017

48. Chen B, Song C, Lin B, et al. A Cross-platform metaverse Data Management System. In: Proceedings of the 2022 IEEE International Conference on Metrology for Extended Reality, Artificial Intelligence and Neural Engineering (MetroXRAINE); 26–28 October 2022; Rome, Italy. pp. 145-150. doi: 10.1109/metroxraine54828.2022.9967588

49. Lo SC, Tsai HH. Design of 3D Virtual Reality in the metaverse for Environmental Conservation Education Based on Cognitive Theory. Sensors. 2022, 22(21): 8329. doi: 10.3390/s22218329

50. Wang Y, Siau KL, Wang L. metaverse and human-computer interaction: A technology framework for 3D virtual worlds. In: Chen JYC, Fragomeni G, Degen H, Ntoa S (editors). HCI International 2022 – Late Breaking Papers: Interacting with eXtended Reality and Artificial Intelligence, Proceedings of the 24th International Conference on Human-Computer Interaction; 26 June 2022–1 July 2022; Online Conference. Springer; 2022. Volume 13518. pp. 213-221. Springer, Cham. https://doi.org/10.1007/978-3-031-21707-4_16

51. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015, 521(7553): 436-444. doi: 10.1038/nature14539

52. Lee HM, Ham SM, Moon H, et al. A metaverse Emotion Mapping System with an AIoT Facial Expression Recognition Device. In: Proceedings of the 2023 IEEE International Conference on metaverse Computing, Networking and Applications (MetaCom). 26–28 June 2023; Kyoto, Japan. pp. 704-707. doi: 10.1109/metacom57706.2023.00132

53. He S, Zhao H, Yu L. The avatar facial expression reenactment method in the metaverse based on overall-local optical-flow estimation and illumination difference. In: Proceedings of the 2023 26th International Conference on Computer Supported Cooperative Work in Design (CSCWD); 24–26 May 2023; Rio de Janeiro, Brazil. pp. 1312-1317. doi: 10.1109/cscwd57460.2023.10152763

54. Jang JY. Analyzing visual behavior of consumers in a virtual reality fashion store using eye tracking. Fashion and Textiles. 2023, 10(1). doi: 10.1186/s40691-023-00345-9

55. Hendrikx M, Meijer S, Van Der Velden J, et al. Procedural content generation for games. ACM Transactions on Multimedia Computing, Communications, and Applications. 2013, 9(1): 1-22. doi: 10.1145/2422956.2422957

56. Kipper G, Rampolla J. Augmented Reality. In: Augmented Reality: An Emerging Technologies Guide to AR, 1st ed. Elsevier; 2013.

57. Furukawa Y, Hernández C. Multi-View Stereo: A Tutorial. Foundations and Trends® in Computer Graphics and Vision. 2015, 9(1-2): 1-148. doi: 10.1561/0600000052

58. Elhagry A. Text-to-metaverse: Towards a Digital Twin-Enabled Multimodal Conditional Generative metaverse. In: Proceedings of the 31st ACM International Conference on Multimedia; 29 October 2023–3 November 2023; Ottawa, ON, Canada. pp. 9336-9339. doi: 10.1145/3581783.3613432

59. Wang X, Chen Q, Li Z. A 3D Reconstruction Method for Augmented Reality Sandbox Based on Depth Sensor. In: Proceedings of the 2021 IEEE 2nd International Conference on Information Technology, Big Data and Artificial Intelligence (ICIBA). 17–19 December 2021; Chongqing, China. pp. 844-849. doi: 10.1109/iciba52610.2021.9687867

60. Wang Y, Su Z, Zhang N, Liu D, et al. A survey on metaverse: Fundamentals, security, and privacy. Available online: https://arxiv.org/abs/2203.02662 (accessed on 2 June 2023).

61. Cheng R, Wu N, Varvello M, et al. Are we ready for metaverse? In: Proceedings of the 22nd ACM Internet Measurement Conference. 25–27 October 2022; New York, United States. pp. 504-518. doi: 10.1145/3517745.3561417

62. Isaac M. Facebook’s new bet on virtual reality: Conference rooms. Available online: https://go.gale.com/ps/i.do?id=GALE%7CA672687966&sid=googleScholar&v=2.1&it=r&linkaccess=abs&issn=22699740&p=AONE&sw=w&userGroupName=anon%7Efcd9bb5b&aty=open-web-entry (accessed on 6 February 2024).

63. Meta. Introducing horizon workrooms: remote collaboration reimagined. Available online: https://www.meta.com/blog/quest/workrooms/ (accessed on 6 February 2024).

64. Shiau WL, Huang LC. Scale development for analyzing the fit of real and virtual world integration: an example of Pokémon Go. Information Technology & People. 2022, 36(2): 500-531. doi: 10.1108/itp-11-2020-0793

65. Niantic Inc. Pockemon go. Available online: https://pokemongolive.com/ (accessed on 2 February 2024).

66. Bonetti F, Perry P. A Review of Consumer-Facing Digital Technologies Across Different Types of Fashion Store Formats. IGI Global; 2017.

67. FashionNetwork. Asos invests in AR technology with new feature. Available online: https://uk.fashionnetwork.com/news/Asos-invests-in-ar-technology-with-new-feature,1109510.html (accessed on 2 June 2023).

68. Egliston B, Carter M. Critical questions for Facebook’s virtual reality: data, power and the metaverse. Internet Policy Review. 2021, 10(4). doi: 10.14763/2021.4.1610

69. Buolamwini J, Gebru T. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Machine Learning Research; 2018.

70. Xu M, Ng WC, Lim WYB, et al. A Full Dive into Realizing the Edge-Enabled metaverse: Visions, Enabling Technologies, and Challenges. IEEE Communications Surveys & Tutorials. 2023, 25(1): 656-700. doi: 10.1109/comst.2022.3221119

71. Shen S, Zhang W. A method for synthesizing dynamic image of virtual human. In: Proceedings of the 2023 3rd International Conference on Consumer Electronics and Computer Engineering (ICCECE). 6–8 January 2023; Guangzhou, China. pp. 698-702. doi: 10.1109/iccece58074.2023.10135229

72. Chang YT. Kernel-wise difference minimization for convolutional neural network compression in metaverse. Available online: https://www.frontiersin.org/articles/10.3389/fdata.2023.1200382/full (accessed on 2 June 2023).

73. Boyer S, Veeramachaneni K. Transfer learning for predictive models in massive open online courses. In: Conati C, Heffernan N, Mitrovic A, Verdejo M (editors). Artificial Intelligence in Education, Proceedings of the 17th International Conference, AIED 2015; 22–26 June 2016; Madrid, Spain. Springer; 2015. Volume 9112. pp. 54-63. doi: 10.1007/978-3-319-19773-9_6

74. Yaqob M, Hafez MM. Metaverse—An overview of daily usage and risks. In: Proceedings of the 2022 OPJU International Technology Conference on Emerging Technologies for Sustainable Development (OTCON); 8–10 February 2023; Raigarh, Chhattisgarh, India. doi: 10.1109/otcon56053.2023.10113922

75. Yosinski J, Clune J, Bengio Y, Lipson H. How transferable are features in deep neural networks? In: Ghahramani Z, Welling M, Cortes C, et al. (editors). Advances in Neural Information Processing Systems 27. Neural Information Processing Systems; 2014.

76. Al-Ghaili AM, Kasim H, Al-Hada NM, et al. A Review of metaverse’s Definitions, Architecture, Applications, Challenges, Issues, Solutions, and Future Trends. IEEE Access. 2022, 10: 125835-125866. doi: 10.1109/access.2022.3225638

77. Zhang L, Qiu Q, Lin H, et al. DreamFace: Progressive Generation of Animatable 3D Faces under Text Guidance. ACM Transactions on Graphics. 2023, 42(4): 1-16. doi: 10.1145/3592094

78. Baltrusaitis T, Ahuja C, Morency LP. Multimodal Machine Learning: A Survey and Taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2019, 41(2): 423-443. doi: 10.1109/tpami.2018.2798607

79. Smart P. Minds in the Metaverse: Extended Cognition Meets Mixed Reality. Philosophy & Technology. 2022, 35(4). doi: 10.1007/s13347-022-00580-w

80. Xu J, Papangelis K, Dunham J, et al. metaverse: The Vision for the Future. In: Proceedings of the CHI’22: CHI Conference on Human Factors in Computing Systems; 29 April 2022; 5 May 2022; New Orleans, LA, USA. pp. 1-3. doi: 10.1145/3491101.3516399

81. Wang P, Wei L, Sun W, et al. Energy-Efficient Distributed Learning and Sharding Blockchain for Sustainable Metaverse. IEEE Wireless Communications. 2023, 30(5): 128-134. doi: 10.1109/mwc.015.2300107

82. Haeberlen A, Phan LTX, McGuire M. metaverse as a Service: Megascale Social 3D on the Cloud. In: Proceedings of the 2023 ACM Symposium on Cloud Computing; 30 October 2023–1 November 2023; Santa Cruz, CA, USA. pp. 298-307. doi: 10.1145/3620678.3624662

83. Brownsword R, Scotford E, Yeung K (editors). Regulating in the Face of Sociotechnical Change. Oxford University Press; 2016. doi: 10.1093/oxfordhb/9780199680832.013.49

84. Riccio P, Oliver N. Racial bias in the beautyverse: Evaluation of augmented-reality beauty filters. Available online: https://link.springer.com/10.1007/978-3-031-25066-8_43 (accessed on 2 June 2023).

85. Belk R, Humayun M, Brouard M. Money, possessions, and ownership in the metaverse: NFTs, cryptocurrencies, Web3 and Wild Markets. Journal of Business Research. 2022, 153: 198-205. doi: 10.1016/j.jbusres.2022.08.031