Bridging deep segmentation and metaverse visualization: Cellpose-based 3D brain tumor reconstruction from MRI

Pranshu Saxena; Aatif Jamshed; Sanjay Kumar Singh; Sandeep Saxena; Sahil Kumar Aggarwal

doi:10.54517/m3727

Publication Frequency

Quarterly (since 2025)

Journal Articles

Search scope

Journal Center

Asia Pacific Academy of Science Pte. Ltd. (APACSCI) specializes in international journal publishing. APACSCI adopts the open access publishing model and provides an important communication bridge for academic groups whose interest fields include engineering, technology, medicine, computer, mathematics, agriculture and forestry, and environment.

more

Volume Arrangement

2025

2024

2023

more

Featured Articles

Garment-aware gaussian for clothed human modeling from monocular video

Reconstructing the human body from monocular video input presents significant challenges, including a limited field of view and difficulty in capturing non-rigid deformations, such as those associated with clothing and pose variations. These challenges often compromise motion editability and rendering quality. To address these issues, we propose a cloth-aware 3D Gaussian splatting approach that leverages the strengths of 2D convolutional neural networks (CNNs) and 3D Gaussian splatting for high-quality human body reconstruction from monocular video. Our method parameterizes 3D Gaussians anchored to a human template to generate posed position maps that capture pose-dependent non-rigid deformations. Additionally, we introduce Learnable Cloth Features, which are pixel-aligned with the posed position maps to address cloth-related deformations. By jointly modeling cloth and pose-dependent deformations, along with compact, optimizable linear blend skinning (LBS) weights, our approach significantly enhances the quality of monocular 3D human reconstructions. We also incorporate carefully designed regularization techniques for the Gaussians, improving the generalization capability of our model. Experimental results demonstrate that our method outperforms state-of-the-art techniques for animatable avatar reconstruction from monocular inputs, delivering superior performance in both reconstruction fidelity and rendering quality.

From paper to virtual: The meta-life of a historical cartographic artifact

The Charta of Rigas Velestinlis is one of the most important works of the eighteenth-century Neo-Hellenic Enlightenment and the most characteristic sample of Greek scholar cartography. Printed in Vienna in 1220 copies in 1796-1797, this emblematic map of the Balkan peninsula significantly influenced the development of ideas and perspectives that inspired the Greek War of Independence from the Ottoman Empire in 1821. The sixty (60) known remaining copies of this valuable material in Greece and abroad remain stored in the confined spaces of libraries, museums, and archives, strictly guarded for security, conservation, and preservation. This renders their access difficult to both the general public and the educational community. Since the Onassis Library and the General State Archives of Greece—Cartographic Heritage Archives both possess an original copy of this historical document each, they design and implement many educational programs aiming at highlighting its importance and reintroducing it to the public. This paper will present how the usage of new technologies, both in software and hardware, has facilitated the showcasing of cultural heritage artifacts, such as Rigas’ Charta, with an emphasis on technologies and resources that are freely available to everyone. It will also be demonstrated how the digitization projects, the digital libraries, repositories, and platforms implemented by many cultural and research organizations during the last decade, presented the opportunity for the new generation to come in contact with a variety of “locked away” historical documents, like Rigas’ Charta, allowing their reuse and reinterpretation while providing unlimited potential for the collection, research and presentation of facts, evidence and data. Furthermore, the incorporation of this digital cultural wealth in the school curriculum through targeted educational programs and the creative combination with open-source metaverse development tools, unleashed the possibilities of reviving the past, extending the life span of old materials to perpetuity. As a result, this multimodal approach paved the way for the emergence of a new more democratic, open-access, and inclusive educational model.

Deciphering avian emotions: A novel AI and machine learning approach to understanding chicken vocalizations

In this groundbreaking study, we present a novel approach to interspecies communication, focusing on the understanding of chicken vocalizations. Leveraging advanced mathematical models in artificial intelligence (AI) and machine learning, we have developed a system capable of interpreting various emotional states in chickens, including hunger, fear, anger, contentment, excitement, and distress. Our methodology employs a cutting-edge AI technique we call Deep Emotional Analysis Learning (DEAL), a highly mathematical and innovative approach that allows for the nuanced understanding of emotional states through auditory data. DEAL is rooted in complex mathematical algorithms, enabling the system to learn and adapt to new vocal patterns over time. We conducted our study with a sample of 80 chickens, meticulously recording and analyzing their vocalizations under various conditions. To ensure the accuracy of our system’s interpretations, we collaborated with a team of eight animal psychologists and veterinary surgeons, who provided expert insights into the emotional states of the chickens. Our system demonstrated an impressive accuracy rate of close to 80%, marking a significant advancement in the field of animal communication. This research not only opens up new avenues for understanding and improving animal welfare but also sets a precedent for further studies in AI-driven interspecies communication. The novelty of our approach lies in its application of sophisticated AI techniques to a largely unexplored area of study. By bridging the gap between human and animal communication, we believe our research will pave the way for more empathetic and effective interactions with the animal kingdom.

Bridging deep segmentation and metaverse visualization: Cellpose-based 3D brain tumor reconstruction from MRI

Pranshu Saxena, Aatif Jamshed, Sanjay Kumar Singh, Sandeep Saxena, Sahil Kumar Aggarwal

Article ID: 3727
Vol 6, Issue 3, 2025

DOI: https://doi.org/10.54517/m3727

Download PDF

Abstract

Accurate and efficient brain tumor segmentation is critical for diagnosis, treatment planning, and outcome monitoring in neuro-oncology. This study presents an integrated framework that combines deep learning-based tumor segmentation with 3D spatial reconstruction and metaverse-aligned visualization. The Cellpose segmentation model, known for its shape-aware adaptability, was applied to grayscale T1-weighted MRI slices to generate binary tumor masks. These 2D masks were reconstructed into 3D surface meshes using the marching cubes algorithm, enabling the computation of clinically relevant spatial parameters including centroid, surface area, bounding box dimensions, and mesh extents. The resulting tumor models were embedded into a global coordinate system and visualized across orthogonal planes, simulating extended reality (XR) environments for immersive anatomical exploration. Quantitative evaluation using DICE, Intersection over Union (IoU), and Positive Predictive Value (PPV) validated the segmentation accuracy, with DICE scores exceeding 0.85 in selected cases. The reconstructed tumors exhibited surface areas ranging from ~45,000 to ~74,000 voxel² units and extended across more than 200 units along the Y and Z axes. Although volumetric values were not computed due to open mesh geometry, the spatial profiles provided a reliable foundation for integration into metaverse platforms. This pipeline offers a lightweight and scalable approach for bridging conventional 2D tumor imaging with immersive 3D applications, paving the way for advanced diagnostic, educational, and surgical planning tools.

Keywords

brain tumor segmentation; cellpose; MRI; 3D reconstruction; marching cubes; metaverse visualization; tumor mesh; medical image analysis

References

1. R. L. Siegel, K. D. Miller, A. Jemal. Cancer statistics, 2023, CA: A Cancer Journal for Clinicians, vol. 73, no. 1, pp. 17–48, 2023. doi: 10.3322/caac.21763.

2. S. Agarwal, A. K. Dohare, P. Saxena,et al. “HDL ACO: Hybrid deep learning and ant colony optimization for ocular optical coherence tomography image classification,” Scientific Reports, vol. 15, p. 5888, Feb. 18, 2025. doi: 10.1038/s41598-025-89961-7.

3. O. Ronneberger, P. Fischer, T. Brox. “U-Net: Convolutional networks for biomedical image segmentation,” in Proc. MICCAI, 2015, pp. 234–241.

4. O. Oktay, J. Schlemper, L. Le Folgoc, et al. “Attention U-Net: Learning where to look for the pancreas,” arXiv preprint, arXiv:1804.03999, 2018.

5. F. Milletari, N. Navab, S. Ahmadi. “V-Net: Fully convolutional neural networks for volumetric medical image segmentation,” in Proc. 3DV, 2016, pp. 565–571.

6. T. Stringer, M. Michaelos, M. Pachitariu. “Cellpose: A generalist algorithm for cellular segmentation,” Nature Methods, vol. 18, pp. 100–106, 2021.

7. W. E. Lorensen, H. E. Cline. “Marching cubes: A high-resolution 3D surface construction algorithm,” ACM SIGGRAPH Computer Graphics, vol. 21, no. 4, pp. 163–169, 1987.

8. B. Meskó, Z. Győrffy. “The role of artificial intelligence in precision medicine and the metaverse,” NPJ Digital Medicine, vol. 5, no. 1, p. 106, 2022.

9. C. Moro, Z. Štromberga, A. Raikos, et al. “The effectiveness of virtual and augmented reality in health sciences and medical anatomy,” Advances in Experimental Medicine and Biology, vol. 1138, pp. 69–76, 2021.

10. B. Amin, R. Sameh Samir, Y. Tarek, et al. “Brain tumor multi classification and segmentation in MRI images using deep learning,” arXiv preprint, arXiv:2304.10039, Apr. 20, 2023. Available: https://arxiv.org/abs/2304.10039.

11. X. Zhang, N. Ou, C. Liu, et al. “Unsupervised brain tumor segmentation with image-based prompts,” arXiv preprint, arXiv:2304.01472, Apr. 4, 2023. Available: https://arxiv.org/abs/2304.01472.

12. T. Ren, E. Honey, H. Rebala, et al. “An optimization framework for processing and transfer learning for the brain tumor segmentation,” arXiv preprint, arXiv:2402.07008, Feb. 10, 2024. doi: https://doi.org/10.48550/arXiv.2402.07008.

13. P. Saxena, S. K. Aggarwal, A. Sinha, et al. “Review of computer assisted diagnosis model to classify follicular lymphoma histology,” Cell Biochemistry & Function, vol. 42, no. 5, e4088, 2024. doi: 10.1002/cbf.4088.

14. A. Hatamizadeh, Y. Tang, D. Yang, et al. “Swin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images,” arXiv preprint, arXiv:2201.01266, Jan. 2022. Available: https://arxiv.org/abs/2201.01266.

15. M. Weber, D. Wild, J. Egger. “Studierfenster: An open science cloud based medical imaging analysis platform,” Journal of Digital Imaging, vol. 35, no. 1, pp. 1–10, Jan. 2022.

16. S. Pieper, M. Halle, R. Kikinis. “3D Slicer,” in Proc. 1st IEEE Int. Symp. Biomed. Imaging: Nano to Macro, 2004, pp. 632–635.

17. R. Chengoden, S. Sankar, M. J. Shankar, et al. “Metaverse for healthcare: A survey on potential applications, challenges and future directions,” arXiv preprint, arXiv:2209.04160, Sep. 2022. Available: https://arxiv.org/abs/2209.04160.

18. P. Saxena, R. Gupta, A. Goel, et al. “Automated Cellular Segmentation in Histopathology: A Multi-Step Mask Prediction Approach for Enhanced Analysis,” 2025 7th International Conference on Signal Processing, Computing and Control (ISPCC), SOLAN, India, 2025, pp. 437-442, doi: 10.1109/ISPCC66872.2025.11039568.

19. F. M. Al-Oqla, N. Nawafleh. “Artificial intelligence and machine learning for additive manufacturing composites toward enriching Metaverse technology,” Metaverse, vol. 5, no. 2, p. 2785, 2024. doi: 10.54517/m.v5i2.2785.

20. L. Zhang. “Editorial: Navigating the convergence of AI and the Metaverse,” Metaverse, vol. 5, no. 2, p. 3219, 2024. doi: 10.54517/m.v5i2.3219.

21. C. O. Yun, T. S. Yun. “Expanding metaverse market: New opportunities and challenges for the content industry,” Metaverse, vol. 5, no. 2, p. 2920, 2024. doi: 10.54517/m.v5i2.2920.

22. N. Kenig, A. Muntaner Vives. “The role of humans in the future of medicine: Completing the cycle,” Metaverse, vol. 6, no. 1, p. 3129, 2025. doi: 10.54517/m.v5i2.m3129.

23. Z. Pan. “Top 10 application scenarios in Metaverse,” Metaverse, vol. 4, no. 1, p. 2202, 2023. doi: 10.54517/m.v4i1.2202.

24. S. Song, X. Chen, J. Liu, et al. “NestedFormer: Nested modality aware transformer for brain tumor segmentation,” Medical Image Analysis, vol. 87, p. 102819, 2023. doi: 10.1016/j.media.2023.102819.

25. J. Guo, K. Liu, B. Zhang, et al. “SegMamba: Long range sequential modeling Mamba for 3D medical image segmentation,” arXiv preprint, arXiv:2402.06738, 2024. Available: https://arxiv.org/abs/2402.06738.

26. Y. Zhao, Y. Wang, Z. Chen, et al. “Serp Mamba: Advancing high resolution retinal vessel segmentation with selective state space model,” arXiv preprint, arXiv:2402.10490, 2024. Available: https://arxiv.org/abs/2402.10490.

27. L. Deng, Y. Wang, Z. Yan, et al. “TP-DRSeg: Improving diabetic retinopathy lesion segmentation with explicit text prompts assisted SAM,” arXiv preprint, arXiv:2402.11312, 2024. Available: https://arxiv.org/abs/2402.11312.

28. P. Saxena, M. Sharma, R. Batra, et al. "XAI for Lymph Node Histopathology: From Explainability to Immersive Metaverse Applications," 2025 7th International Conference on Signal Processing, Computing and Control (ISPCC), SOLAN, India, 2025, pp. 304-309, doi: 10.1109/ISPCC66872.2025.11039319.

29. A. Hatamizadeh, Y. Tang, D. Yang, et al. “Medical open network for AI (MONAI),” arXiv preprint, arXiv:2012.05776, Dec. 2020. Available: https://arxiv.org/abs/2012.05776.

30. T. Huynh-The, Q. V. Pham, X. Q. Pham, et al. “Artificial intelligence for the metaverse: A survey,” Engineering Applications of Artificial Intelligence, vol. 117, p. 105581, 2023. doi: 10.1016/j.engappai.2022.105581.

31. R. Chengoden, S. Sankar, M. J. Shankar, et al. “Metaverse for healthcare: A survey on potential applications, challenges and future directions,” Biomedical Signal Processing and Control, vol. 71, p. 103174, 2022. doi: 10.1016/j.bspc.2021.103174.

32. J. Wang. “Application of Brain–Computer Interface Technology in the Metaverse: Challenges and Future Directions,” Journal of Intelligent & Fuzzy Systems, vol. 45, no. 4, pp. 4959–4967, 2023. doi: 10.3233/JIFS-230242.

33. A. Drigas, E. Sideraki. “Neuroplasticity and Cognitive Enhancement in Virtual Reality Environments: A Road to the Metaverse,” International Journal of Online and Biomedical Engineering (iJOE), vol. 20, no. 1, pp. 4–16, 2024. doi: 10.3991/ijoe.v20i01.40479.

Supporting Agencies

This work is licensed under a Creative Commons Attribution 4.0 International License.

This site is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).

Academic Supporter

Institute for Metaverse, School of Artificial Intelligence, Nanjing University of Information Science & Technology (NUIST), China

Editor-in-Chief

Prof. Zhigeng Pan

Director, Institute for Metaverse, Nanjing University of Information Science & Technology, China

Honorary Editor-in-Chief

Prof. Jianrong Tan

Academician, Chinese Academy of Engineering, China

Indexing & Archiving

News & Announcements

2025-05-20

Call for papers for SIGGRAPH ASIA 2025!

Conference Time

December 15-18, 2025

Conference Venue

Hong Kong Convention and Exhibition Center (HKCEC)

...

2025-04-23

Metaverse Scientist Forum No.3

Metaverse Scientist Forum No.3 was successfully held on April 22, 2025, from 19:00 to 20:30 (Beijing Time)...

2025-04-21

Congratulations to Metaverse on being indexed in Scopus!

We received the Scopus notification on April 19th, confirming that the journal has been successfully indexed by Scopus...

2025-04-15

Updated Submission Guidelines for Manuscript Figures!

We are pleased to announce that we have updated the requirements for manuscript figures in the submission guidelines. Manuscripts submitted after April 15, 2025 are required to strictly adhere to the change. These updates are aimed at ensuring the highest quality of visual content in our publications and enhancing the overall readability and impact of your research. For more details, please find it in sumissions...

More Announcements...

Article Tools

Indexing metadata

How to cite item

Review policy