Open Journal Systems

Virtual reality painting: A structured review of a decade of innovation

Yan Shi, Yiwen Lu, Lijuan Liu, Enmao Liu

Article ID: 2546
Vol 5, Issue 1, 2024

DOI: https://doi.org/10.54517/m.v5i1.2546
VIEWS - 29 (Abstract)

Abstract

Various studies have widely utilized virtual reality (VR) in painting creation. However, there has been few conclusive studies on VR painting. The present systematic literature review describes the current state of knowledge and practices using VR in painting and provides guidance for painters and scholars focusing on human computer interaction design by critically appraising and summarizing the existing research. It also outlines a wide range of results yielded by quantitative, qualitative and mixed-method studies to investigate potential benefits, difficulties, and effectiveness of VR painting. Overall, 44 studies published from 2013 until 2023 were analysed, with 16 articles introduced the hardware technology and interaction mode of VR painting, 17 articles introduced the beautification method of VR painting, and 11 articles focused on the application of VR painting in different fields. Implications for practice and research are discussed in detail, as the hardware selection, interaction, and physical simulation in VR painting scenes.

Keywords

virtual reality; painting; sketch; human-computer interaction; systematic review

Full Text:

PDF

References

1. Liang Z, Yi J, Yuanxia L, et al. Virtual reality (VR) technology and development review. China Computer & Communication. 2019; 31(17): 126-128.

2. Sato R. Innovative development of visual arts based on VR technology. Journal of Jilin College of the Arts. 2018; 2: 22-26.

3. Zhihong Z. VR images: Material media, generation—Body and idea. Journal of Beijing Film Academy. 2021; 11: 51-62.

4. Chunlei C, Yang Y, Zheyu F, et al. Research and Prospect of “meta” design: Digital content design in the age of meta Universe. Packaging Engineering. 2023; 44(4): 59-68,86,14. doi: 10.19554/j.cnki.1001-3563.2023.04.008

5. Blanchet C. Sandde: The World’s First Freehand S-3D Animation System. Available online: https://www.ada-x.org/en/activities/sandde-the-worlds-first-freehand-s-3d-animation-system/ (accessed on 2 March 2023).

6. Keefe DF, Feliz DA, Moscovich T, et al. CavePainting: A fully immersive 3D artistic medium and interactive experience. Proceedings of the 2001 symposium on Interactive 3D graphics. 2001; 85-93.

7. Tilt Brush. Available online: https://www.tiltbrush.com/ (accessed on 2 March 2023).

8. Oculus Quill. Available online: https://quill.fb.com/ (accessed on 2 March 2023).

9. Gravitysketch. Available online: https://www.gravitysketch.com/ (accessed on 2 March 2023).

10. Gesture VR. Available online: https://www.gesture-vr.com/ (accessed on 2 March 2023).

11. Türkmen R, Pfeuffer K, Barrera Machuca MD, et al. Exploring discrete drawing guides to assist users in accurate mid-air sketching in VR. In: Proceedings of the CHI Conference on Human Factors in Computing Systems Extended Abstracts; 29 April–5 May 2022; New Orleans, LA, USA. pp. 1-7.

12. Hoffmann PP, Elsayed H, Mühlhäuser M, et al. ThermalPen: Adding Thermal Haptic Feedback to 3D Sketching. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems; 23–28 April 2023; Hamburg, Germany. pp. 1-4.

13. Zeidler D, McGinity M. Bodylab. Proceedings of the ACM on Computer Graphics and Interactive Techniques. 2023; 6(2): 1-12. doi: 10.1145/3597631

14. Qinping Z. Virtual reality overview. SCIENTIA SINICA Informationis. 2009, 39(1): 2-46.

15. Hoskinson R, Naugle E. A mobile head-mounted display for action sports. In: Proceedings of the 2012 Third FTRA International Conference on Mobile, Ubiquitous, and Intelligent Computing; 26–28 June 2012; Vancouver, BC, Canada. pp. 1-4.

16. Pham DM, Stuerzlinger W. Is the pen mightier than the controller? A comparison of input devices for selection in virtual and augmented reality. In: Proceedings of the 25th ACM Symposium on Virtual Reality Software and Technology; 12–15 November 2019; Parramatta, NSW, Australia. pp. 1-11.

17. Elsayed H, Barrera Machuca MD, Schaarschmidt C, et al. Vrsketchpen: Unconstrained haptic assistance for sketching in virtual 3d environments. In: Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology; 1–4 November 2020; Canada. pp. 1-11.

18. Arora R, Kazi RH, Anderson F, et al. Experimental Evaluation of Sketching on Surfaces in VR. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems; 6–11 May 2017; USA. pp. 5643-5654.

19. An SG, Kim Y, Lee JH, et al. Collaborative experience prototyping of automotive interior in VR with 3D sketching and haptic helpers. In: Proceedings of the 9th international conference on automotive user interfaces and interactive vehicular applications; 24–27 September 2017; Oldenburg, Germany. pp. 183-192.

20. Bae SH, Balakrishnan R, Singh K. ILoveSketch: as-natural-as-possible sketching system for creating 3d curve models. In: Proceedings of the 21st annual ACM symposium on User interface software and technology; 19–22 October 2008; Monterey, CA, USA. pp. 151-160.

21. Bae SH, Balakrishnan R, Singh K. EverybodyLovesSketch: 3D sketching for a broader audience. In: Proceedings of the 22nd annual ACM symposium on User interface software and technology; 4–7 October 2009; Victoria, BC, Canada. pp. 59-68.

22. Drey T, Gugenheimer J, Karlbauer J, et al. VRsketchin: Exploring the design space of pen and tablet interaction for 3d sketching in virtual reality. In: Proceedings of the 2020 CHI conference on human factors in computing systems; 25–30 April 2020; Honolulu, HI, USA. pp. 1-14.

23. Gasques D, Johnson JG, Sharkey T, et al. Pintar: Sketching spatial experiences in augmented reality. In: Proceedings of the Companion Publication of the 2019 on Designing Interactive Systems Conference 2019 Companion; 23–28 June 2019; San Diego, CA, USA. pp. 17-20.

24. Gasques D, Johnson JG, Sharkey T, et al. What you sketch is what you get: Quick and easy augmented reality prototyping with pintar. In: Proceedings of the Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems; 4–9 May 2019; Glasgow, Scotland, UK. pp. 1-6.

25. Arora R, Habib Kazi R, Grossman T, et al. Symbiosissketch: Combining 2d & 3d sketching for designing detailed 3d objects in situ. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems; 21–26 April 2018; Montreal, QC, Canada. pp. 1-15.

26. Son J, Sra M. Exploring Emotion Brushes for a Virtual Reality Painting Tool. In: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology; 8–10 December 2021; Osaka, Japan. pp. 1-3.

27. Shi R, Zhang J, Yue Y, et al. Exploration of Bare-Hand Mid-Air Pointing Selection Techniques for Dense Virtual Reality Environments. In: Proceedings of the Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems; 23-28 April 2023; Hamburg, Germany. pp. 1-7.

28. Jiang Y, Zhang C, Fu H, et al. Handpainter-3d Sketching in VR with Hand-based Physical Proxy. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systemsl 8-13 May 2021; Yokohama, Japan. pp. 1-13.

29. Kim Y, An SG, Lee JH, et al. Agile 3D sketching with air scaffolding. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems; 21-26 April 2018; Montreal, QC, Canada. pp. 1-12.

30. Bohari U, Chen TJ, Vinayak. To draw or not to draw: recognizing stroke-hover intent in non-instrumented gesture-free mid-air sketching. In: Proceedings of the 23rd International Conference on Intelligent User Interfaces; 7-11 March 2018; Tokyo, Japan. pp. 177-188.

31. Zhang X, Jing L, Yuan L, et al. Balance Design of Immersion and Comfort in Virtual Reality Technology. Packaging Engineering. 2021; 42(12): 91-97. doi: 10.19554/j.cnki.1001-3563.2021.12.012

32. Panagiotopoulou E, Filippetti ML, Tsakiris M, et al. Affective Touch Enhances Self-Face Recognition During Multisensory Integration. Scientific Reports. 2017; 7(1). doi: 10.1038/s41598-017-13345-9

33. Li Z, Chen P, Zhang H, et al. Designing a Smart VR Painting System with Multisensory Interaction for Immersive Experience. In: Proceedings of the 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW); 1 May 2023; Shanghai, China. pp. 711-712.

34. Ranasinghe N, Jain P, Thi Ngoc Tram N, et al. Season traveller: Multisensory narration for enhancing the virtual reality experience. In: Proceedings of the 2018 CHI conference on human factors in computing systems; 21-26 April 2018; Montreal, QC, Canada. pp. 1-13.

35. Reiter K, Pfeuffer K, Esteves A, et al. Look & turn: One-handed and expressive menu interaction by gaze and arm turns in VR. In: Proceedings of the 2022 Symposium on Eye Tracking Research and Applications; 8-11 June 2022; Seattle, WA, USA. pp. 1-7.

36. Wolf E, Klüber S, Zimmerer C, et al. “Paint that object yellow”: Multimodal Interaction to Enhance Creativity During Design Tasks in VR. In: Proceedings of the 2019 International Conference on Multimodal Interaction; 14-18 October 2019; Suzhou, China. pp. 195-204.

37. Mäkelä W, Reunanen M, Takala T. Possibilities and limitations of immersive free-hand expression: a case study with professional artists. In: Proceedings of the 12th annual ACM international conference on Multimedia; 10-16 October 2004; New York, NY, USA. pp. 504-507.

38. Mäkelä W, Ilmonen T. Drawing, painting and sculpting in the air: Development studies about an immersive free-hand interface for artists. In: Proceedings of the Beyond Wand and Glove Based Interaction Workshop; 28 March 2004; Chicago, USA. pp. 89-92.

39. Kang J, Wang S, Wang S, et al. Fluid3dguides: A technique for structured 3d drawing in VR. In: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology; 8-10 December 2021; Osaka, Japan. pp. 1-2.

40. Machuca MDB, Stuerzlinger W, Asente P. Smart3dguides: Making unconstrained immersive 3d drawing more accurate. In: Proceedings of the 25th ACM Symposium on Virtual Reality Software and Technology; 12-15 November 2019; Parramatta, NSW, Australia. pp. 1-13.

41. Machuca MDB, Asente P, Stuerzlinger W, et al. Multiplanes: Assisted freehand VR sketching. In: Proceedings of the 2018 ACM Symposium on Spatial User Interaction; 13-14 October 2018; Berlin, Germany. pp. 36-47.

42. Türkmen R, Pfeuffer K, Barrera Machuca MD, et al. Exploring discrete drawing guides to assist users in accurate mid-air sketching in VR. In: Proceedings of the CHI Conference on Human Factors in Computing Systems Extended Abstracts; 29 April 2022–5 May 2022; New Orleans, LA, USA. pp. 1-7.

43. Kusunoki M, Furuhama R, Toshima R, et al. MultiBrush: 3D Brush Painting Using Multiple Viewpoints in Virtual Reality. In: Proceedings of the 2023 9th International Conference on Virtual Reality (ICVR); 12-14 May 2023; Xianyang, China. pp. 481-486.

44. Fišer J, Asente P, Sýkora D. ShipShape: a drawing beautification assistant. In: Proceedings of the workshop on Sketch-Based Interfaces and Modeling; 20 June 2015. pp. 49-57.

45. Kwan KC, Fu H. Mobi3dsketch: 3d sketching in mobile AR. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems; 4-9 May 2019; Glasgow, Scotland, UK. pp. 1-11.

46. Barrera Machuca MD, Asente P, Lu J, et al. Multiplanes: Assisted freehand VR drawing. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology; 22-25 October 2017; Québec City, QC, Canada. pp. 1-3.

47. Son J. emoPaint: Exploring Emotion and Art in a VR-based Creativity Tool. In: Proceedings of 2021 9th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW); 10 January 2022; Nara, Japan. pp. 1-5.

48. Otsuki M, Sugihara K, Toda A, et al. A brush device with visual and haptic feedback for virtual painting of 3D virtual objects. Virtual Reality. 2018; 22: 167-181.

49. Sun L, Xiang W, Chai C, et al. Designers’ perception during sketching: An examination of Creative Segment theory using eye movements. Design Studies. 2014; 35(6): 593-613. doi: 10.1016/j.destud.2014.04.004

50. Wacker P, Wagner A, Voelker S, et al. Physical guides: An analysis of 3d sketching performance on physical objects in augmented reality. In: Proceedings of the 2018 ACM Symposium on Spatial User Interaction; 13-14 October 2018; Berlin, Germany. pp. 25-35.

51. Pfeuffer K, Mecke L, Delgado Rodriguez S, et al. Empirical evaluation of gaze-enhanced menus in virtual reality. In: Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology; 1-4 November 2020; Canada. pp. 1-11.

52. Zhao M, Jin L. A Re-presentation of the Chinese Traditional Art by the VR Technology in the Digital Age: Taking Mogao Grottoes in Dunhuang as an Example. Journal of Physics: Conference Series. 2021; 1961(1): 012075. doi: 10.1088/1742-6596/1961/1/012075

53. González-López MJ. Virtual reintegration of lacunae. The paintings of the Hall of the Kings of the Alhambra in Granada (Spanish). PH: Boletín del Instituto Andaluz del Patrimonio Histórico. 2009; 17(70): 98-113.

54. Pavlidis G, Koutsoudis A, Arnaoutoglou F, et al. Methods for 3D digitization of cultural heritage. Journal of Cultural Heritage. 2007; 8(1): 93-98. doi: 10.33349/2009.70.2754

55. Soto-Martin O, Fuentes-Porto A, Martin-Gutierrez J. A digital reconstruction of a historical building and virtual reintegration of mural paintings to create an interactive and immersive experience in virtual reality. Applied Sciences. 2020; 10(2): 597.

56. Fu X, Zhu Y, Xiao Z, et al. RestoreVR: generating embodied knowledge and situated experience of dunhuang mural conservation via interactive virtual reality. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems; 25-30 April 2020; Honolulu, HI, USA. pp. 1-13.

57. Alice S. VR Artist Base in Beijing. Available online: https://www.qiemamaalice.com/ (accessed on 2 March 2023).

58. Quilez I. Quill: VR drawing in the production of Oculus Story Studio’s new movie. In: Proceedings of the ACM SIGGRAPH 2016 Real-Time Live; 24-28 July 2016; Anaheim, California. pp. 26.

59. Cutler L, Darnell E, Dirksen N, et al. From quest to quill: pushing the boundaries of VR storytelling in baobab’s Baba Yaga and Namoo. In: Proceedings of the ACM SIGGRAPH 2021 Production Sessions; 9-13 August 2021; Virtual, Event, USA. pp. 1-1.

60. Qin L, Jun T. Research on the creative method of transforming traditional painting into VR animation—taking Riverside Scene at Qingming Festival as an example. Art Education Research. 2022; (12): 89-91.

61. Baron L, Wang Q, Segear S, et al. Enjoyable physical therapy experience with interactive drawing games in immersive virtual reality. In: Proceedings of the 2021 ACM Symposium on Spatial User Interaction; 9-10 November 2021; USA. pp. 1-8.

62. Bell IH, Nicholas J, Alvarez-Jimenez M, et al. Virtual reality as a clinical tool in mental health research and practice. Dialogues in Clinical Neuroscience. 2020; 22(2): 169-177. doi: 10.31887/DCNS.2020.22.2/lvalmaggia

63. Kothgassner OD, Goreis A, Kafka JX, et al. Virtual reality exposure therapy for posttraumatic stress disorder (PTSD): A meta-analysis. European Journal of Psychotraumatology. 2019; 10(1): 1654782. doi: 10.1080/20008198.2019.1654782

64. Tarrant J, Viczko J, Cope H. Virtual reality for anxiety reduction demonstrated by quantitative EEG: A pilot study. Frontiers in Psychology. 2018; 9: 1280. doi: 10.3389/fpsyg.2018.01280

65. Chan SHM, Qiu L, Esposito G, et al. Nature in virtual reality improves mood and reduces stress: Evidence from young adults and senior citizens. Virtual Reality. 2023; 27(4): 3285-3300. doi: 10.1007/s10055-021-00604-4

66. Richesin MT, Baldwin D R, Wicks L A M. Art making and virtual reality: A comparison study of physiological and psychological outcomes. The Arts in Psychotherapy. 2021, 75: 101823. https://doi.org/10.1016/j.aip.2021.101823

67. Kaimal G, Carroll-Haskins K, Ramakrishnan A, et al. Outcomes of visual self-expression in virtual reality on psychosocial well-being with the inclusion of a fragrance stimulus: A pilot mixed-methods study. Frontiers in Psychology. 2020; 11: 589461. doi: 10.3389/fpsyg.2020.589461

68. Tan J, Kannis-Dymand L, Jones C. Examining the potential of VR program Tilt Brush in reducing anxiety. Virtual Reality. 2023; 27(4): 3379-3391. doi: 10.1007/s10055-022-00711-w

69. Li N, Han T, Tian F, et al. Get a grip: Evaluating grip gestures for VR input using a lightweight pen. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems; 25-30 April 2020; Honolulu, HI, USA. pp. 1-13.

70. Hennessey JW, Liu H, Winnemöller H, et al. How2Sketch: Generating easy-to-follow tutorials for sketching 3D objects. arXiv preprint arXiv:1607.07980. 2016. doi: 10.48550/arXiv.1607.07980

71. Knispel J, Bullock F. Collaborative VR painting in web browsers. In: Proceedings of the SIGGRAPH Asia 2017 VR Showcase (SA ‘17). Association for Computing Machinery; New York, NY, USA. pp. 1–2.

72. Lu Z, Shi Y, Tao R, et al. Blockchain for digital rights management of design works. In: Proceedings of the 2019 IEEE 10th international conference on software engineering and service science (ICSESS); 18-20 October 2019; Beijing, China. pp. 596-603.