Using a Mobile Phone in VR

Virtual Reality (VR) Head-Mounted Display (HMD) technology immerses a user in a computer generated virtual environment. However, a VR HMD also blocks the users’ view of their physical surroundings, and so prevents them from using their mobile phones in a natural manner. In this project, we present a novel Augmented Virtuality (AV) interface that enables people to naturally interact with a mobile phone in real time in a virtual environment. The system allows the user to wear a VR HMD while seeing his/her 3D hands captured by a depth sensor and rendered in different styles, and enables the user to operate a virtual mobile phone aligned with their real phone.

Project Video(s):
https://www.youtube.com/watch?v=pBDRl4WRnxk

People

Li Zhang
Mark Billinghurst
Huidong Bai

Publications

  • Bringing full-featured mobile phone interaction into virtual reality
    H Bai, L Zhang, J Yang, M Billinghurst

    Bai, H., Zhang, L., Yang, J., & Billinghurst, M. (2021). Bringing full-featured mobile phone interaction into virtual reality. Computers & Graphics, 97, 42-53.

    BibTexLinksAbstract
    @article{bai2021bringing,
    title={Bringing full-featured mobile phone interaction into virtual reality},
    author={Bai, Huidong and Zhang, Li and Yang, Jing and Billinghurst, Mark},
    journal={Computers \& Graphics},
    volume={97},
    pages={42--53},
    year={2021},
    publisher={Elsevier}
    }
    View: https://doi.org/10.1016/j.cag.2021.04.004
    Video: https://www.youtube.com/watch?v=pBDRl4WRnxk

    Virtual Reality (VR) Head-Mounted Display (HMD) technology immerses a user in a computer generated virtual environment. However, a VR HMD also blocks the users’ view of their physical surroundings, and so prevents them from using their mobile phones in a natural manner. In this paper, we present a novel Augmented Virtuality (AV) interface that enables people to naturally interact with a mobile phone in real time in a virtual environment. The system allows the user to wear a VR HMD while seeing his/her 3D hands captured by a depth sensor and rendered in different styles, and enables the user to operate a virtual mobile phone aligned with their real phone. We conducted a formal user study to compare the AV interface with physical touch interaction on user experience in five mobile applications. Participants reported that our system brought the real mobile phone into the virtual world. Unfortunately, the experiment results indicated that using a phone with our AV interfaces in VR was more difficult than the regular smartphone touch interaction, with increased workload and lower system usability, especially for a typing task. We ran a follow-up study to compare different hand visualizations for text typing using the AV interface. Participants felt that a skin-colored hand visualization method provided better usability and immersiveness than other hand rendering styles.