Adaptive Instructor Aid for Virtual Reality/Augmented Reality Enabled Classroom Training

OUSD (R&E) CRITICAL TECHNOLOGY AREA(S): Advanced Computing and Software OBJECTIVE: Develop a system for real-time monitoring of student performance and performance-driven instructional adaptation within an immersive (that is, virtual or augmented reality enabled) training experience. DESCRIPTION: Virtual Reality (VR) and Augmented Reality (AR) are becoming more common in Navy training as instructional media that provide visually immersive training conditions, both in the classroom for individual training and in team dynamic settings. This has the advantage of exposing the student to visual operational conditions that are motivationally immersive and hard to replicate or train to in purely live training. While VR and AR technologies are modern and alluring, use of these technologies alone does not ensure training effectiveness. Furthermore, their introduction may insulate an instructor from fully observing and adaptively interacting with the student, reducing natural and traditional student/instructor experiences. A body of knowledge in basic research addresses the effective use of VR/AR, including the need to monitor student performance and adapting the pace or content of instructional material based on this [Ref 1]. This STTR topic seeks to develop a VR/AR learning system that adapts to the student based on the competency of the student's performance and yields a learning gain that is significantly better than current training methodologies. Toward this end, the aim is to develop a working prototype and a student performance measurement approach that can be generalized across a range of classroom adaptive training environments where VR and AR could be applied. PHASE I: Determine a technical approach to monitor competency of student performance while using wearable VR or AR instructional media and use such information to inform and adapt instructional content. Determine the metrics to measure student performance shortfalls, and for assessing system training effectiveness. Include designs of baseline measures that would be administered before training and a comparison measure after introducing this new approach, using any current Navy Use Case. Phase I accomplishments will be presented to cognizant ONR Program Officers for feedback before initiating Phase II activity. PHASE II: Apply the Phase I approach to a representative Navy classroom training environment where VR and/or AR are, or could be, used. Implement the technical approach in a prototype and collect metrics for baseline comparison. The government will provide a representative Navy VR/AR classrooms environment if the performing team does not have access to or insight into the content and design of Navy-relevant classroom settings. PHASE III DUAL USE APPLICATIONS: Assess the training value of the approach developed in Phase II, comparing it with VR or AR-based training without the assessment and adaptivity of this approach. If the approach developed is demonstrated to be significantly better in training efficacy (speed and/or level of the students' content mastery) based on criteria agreed on by ONR and Fleet representatives, demonstrate the system to leadership in the Fleet Training Wholeness Program of Record. Commercialization opportunities should be explored in Phase II and pursued in Phase III. Potential markets that would be interested in this technology include aviation, maintenance, law enforcement, and the medical field. REFERENCES: Mayer, R.E., Makransky, G., & Parong, J. (In press). The promise and pitfalls of learning in immersive virtual reality. International Journal of Human-computer Interaction, Volume 39, 2023 - Issue 11: Trends in Adaptive Interactive Training Systems. http://www.tandfonline.com/doi/abs/10.1080/10447318.2022.2108563?journalCode=hihc20 Landsberg, C.R., Mercado, A.D., Van Buskirk, W.L., Lineberry, M. & Steinhauser, N. (2021). Evaluation of an adaptive training system for submarine periscope operations. Proceedings of the Human Factors and Ergonomics Society. 56(1), 2422-2426. Metzler-Baddeley, C., & Baddeley, R. J. (2009). Does adaptive training work?. Applied Cognitive Psychology, 23, (2), 254-266. KEYWORDS: Virtual reality; augmented reality; Adaptive training; Instructional media; immersive environments; classroom training