Teaching WebAR development with integrated machine learning: a methodology for immersive and intelligent educational experiences

Abstract

Augmented reality (AR) and machine learning (ML) are rapidly growing technologies with immense potential for transforming education. Web-based augmented reality (WebAR) provides a promising approach to delivering immersive learning experiences on mobile devices. Integrating machine learning models into WebAR applications can enable advanced interactive effects by responding to user actions, thus enhancing the educational content. However, there is a lack of effective methodologies to teach students WebAR development with integrated machine learning. This paper proposes a methodology with three main steps: (1) Integrating standard TensorFlow.js models like handpose into WebAR scenes for gestures and interactions; (2) Developing custom image classification models with Teachable Machine and exporting to TensorFlow.js; (3) Modifying WebAR applications to load and use exported custom models, displaying model outputs as augmented reality content. The proposed methodology is designed to incrementally introduce machine learning integration, build an understanding of model training and usage, and spark ideas for using machine learning to augment educational content. The methodology provides a starting point for further research into pedagogical frameworks, assessments, and empirical studies on teaching WebAR development with embedded intelligence.