Tangible Augmented Reality for Learning Programming Learning

This project explores how tangible Augmented Reality (AR) can be used to teach computer programming. We have developed TARPLE, A Tangible Augmented Reality Programming Learning Environment, and are studying its efficacy for teaching text-based programming languages to novice learners. TARPLE uses physical blocks to represent programming functions and overlays virtual imagery on the blocks to show the programming code. Use can arrange the blocks by moving them with their hands, and see the AR content either through the Microsoft Hololens2 AR display, or a handheld tablet.

This research project expands upon the broader question of educational AR as well as on the questions of tangible programming languages and tangible learning mediums. When supported by the embodied learning and natural interaction affordances of AR, physical objects may hold the key to developing fundamental knowledge of abstract, complex subjects for younger learners in particular. It may also serve as a powerful future tool in advancing early computational thinking skills in novices. Evaluation of such learning environments addresses the hypothesis that hybrid tangible AR mediums are able to support an extended learning taxonomy both within the classroom and without.

We are currently conducting user studies to explore how learning with TARPLE is different from traditional programming tutorials, and the benefits that tangible AR could bring to an educational environment. The initial target for the application is high school children with little experience of programming, but the project could be expanded to support a wide range of other people who are programming novices.

Project Video(s):
https://youtu.be/tOlavOCUv7s

People

Dmitry Resnyansky
Mark Billinghurst
Arindam Dey
Arindam Dey

Publications

  • A prototype system with AR-supported TUI being used in a code construction scenario. Registration patterns on tangible code blocks are tracked with an on-board camera from a handheld tablet device, and respond by displaying passive augmentation to indicate potential for interaction (respective bounding box on each block), as well as corresponding augmented annotation depending on proximity to the viewer. Additional text cues are output to the system's HUD (seen below the main workspace area) to guide learner's programming activity within the environment.
    Augmented reality-supported tangible gamification for debugging learning
    Dmitry Resnyansky

    Resnyansky, D. (2020, December). Augmented reality-supported tangible gamification for debugging learning. In 2020 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE) (pp. 377-383). IEEE.

    BibTexLinksAbstract
    @INPROCEEDINGS{9368410,
    author={Resnyansky, Dmitry},
    booktitle={2020 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE)},
    title={Augmented Reality-Supported Tangible Gamification for Debugging Learning},
    year={2020},
    volume={},
    number={},
    pages={377-383},
    doi={10.1109/TALE48869.2020.9368410}}
    View: https://ieeexplore.ieee.org/document/9368410
    Innovative technologies such as Augmented Reality (AR) and Virtual Reality, tangible user interfaces (TUIs), computer games, robotics and microprocessors have seen an interest within educational research due to their potential for fostering active learning in and outside the classroom. This paper aims to explore the affordances of AR and TUIs as mediums of instruction to address the problem of teaching and learning text-based computer languages, computer science concepts, and programming skills such as debugging. It presents parallels between the technology-supported learning and the active, scaffolded narrative entertainment experience in videogames, and suggests a conceptual framework for the design of learning environments for programming and debugging by using AR and tangible interaction to support scaffolding.
  • Using an (a) explanation, (b) example, or (c) hint helper block, a brief summary of the code component, examples of its usage, and a guide as to the succeeding component in the statement may respectively be displayed next to the original syntax through 3D text annotation and graphical representation.
    An AR/TUI-supported Debugging Teaching Environment
    Dmitry Resnyansky , Mark Billinghurst , Arindam Dey

    Resnyansky, D., Billinghurst, M., & Dey, A. (2019, December). An AR/TUI-supported debugging teaching environment. In Proceedings of the 31st Australian Conference on Human-Computer-Interaction (pp. 590-594).

    BibTexLinksAbstract
    @inproceedings{10.1145/3369457.3369538,
    author = {Resnyansky, Dmitry and Billinghurst, Mark and Dey, Arindam},
    title = {An AR/TUI-Supported Debugging Teaching Environment},
    year = {2020},
    isbn = {9781450376969},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3369457.3369538},
    doi = {10.1145/3369457.3369538},
    abstract = {This paper presents research on the potential application of Tangible and Augmented Reality (AR) technology to computer science education and the teaching of programming in tertiary settings. An approach to an AR-supported debugging-teaching prototype is outlined, focusing on the design of an AR workspace that uses physical markers to interact with content (code). We describe a prototype which has been designed to actively scaffold the student's development of the two primary abilities necessary for effective debugging: (1) the ability to read not just the code syntax, but to understand the overall program structure behind the code; and (2) the ability to independently recall and apply the new knowledge to produce new, working code structures.},
    booktitle = {Proceedings of the 31st Australian Conference on Human-Computer-Interaction},
    pages = {590–594},
    numpages = {5},
    keywords = {tangible user interface, tertiary education, debugging, Human-computer interaction, augmented reality},
    location = {Fremantle, WA, Australia},
    series = {OzCHI '19}
    }
    View: https://dl.acm.org/doi/abs/10.1145/3369457.3369538
    This paper presents research on the potential application of Tangible and Augmented Reality (AR) technology to computer science education and the teaching of programming in tertiary settings. An approach to an AR-supported debugging-teaching prototype is outlined, focusing on the design of an AR workspace that uses physical markers to interact with content (code). We describe a prototype which has been designed to actively scaffold the student's development of the two primary abilities necessary for effective debugging: (1) the ability to read not just the code syntax, but to understand the overall program structure behind the code; and (2) the ability to independently recall and apply the new knowledge to produce new, working code structures.
  • TUI/AR-based teaching programming tools
    The potential of augmented reality for computer science education
    Dmitry Resnyansky; Emin İbili; Mark Billinghurst

    Resnyansky, D., Ibili, E., & Billinghurst, M. (2018, December). The potential of augmented reality for computer science education. In 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE) (pp. 350-356). IEEE.

    BibTexLinksAbstract
    @INPROCEEDINGS{8615331,
    author={Resnyansky, Dmitry and İbili, Emin and Billinghurst, Mark},
    booktitle={2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE)},
    title={The Potential of Augmented Reality for Computer Science Education},
    year={2018},
    volume={},
    number={},
    pages={350-356},
    doi={10.1109/TALE.2018.8615331}}
    View: https://ieeexplore.ieee.org/abstract/document/8615331
    Innovative approaches in the teaching of computer science are required to address the needs of diverse target audiences, including groups with minimal mathematical background and insufficient abstract thinking ability. In order to tackle this problem, new pedagogical approaches that make use of technologies such as Virtual and Augmented Reality, Tangible User Interfaces, and 3D graphics are needed. This paper draws upon relevant pedagogical and technological literature to determine how Augmented Reality can be more fully applied to computer science education.