Mixed Reality HeartMR
Research development
Challenge
The HeartMR project sought to transform educational learning methods in anatomy. The challenge was twofold: upgrading the existing JavaScript application to C# for more advanced AR/VR capabilities, and creating an engaging, interactive learning environment. The goal was to improve knowledge retention through hands-on virtual experiences and to provide teachers and researchers with tools to assess and enhance learning methods.
Solution
We upgraded the HeartMR application to a C# codebase in Unity 3D, integrating Vuforia for augmented reality. The app features a 3D heart model for study and test modes, screen recording of test attempts for playback and analysis, and a results dashboard built with PHP and JavaScript. This solution provides an engaging and comprehensive learning experience, allowing students to interact with and understand cardiac anatomy in a virtual environment.
Summary
HeartMR, a collaborative project led by Associate Professor Michael Cowling from CQUniversity and Dr. James Birt form Bond University, represents a significant advancement in the use of mixed reality for educational purposes.
Challenges: This project aimed to upgrade an existing Unity 3D application from JavaScript to C# to leverage advanced AR/VR technologies. The primary challenge was creating an interactive and immersive learning environment for cardiac anatomy, enhancing both the understanding and retention of complex information.
Technical Enhancements: The application was redeveloped in C# with Unity 3D, incorporating the Vuforia AR library. A 3D heart model was mapped to a physical cube marker, enabling users to interact with the model in a virtual space, enhancing the learning experience.
Interactive Learning Tools:
- Study and Test Modes: Allows students to learn and test their knowledge of heart anatomy.
- Screen Recording: Captures test attempts for review, facilitating learning and problem-solving analysis in VR/AR environments. This feature was implemented by capturing images at intervals, compressing them into videos using FFmpeg, and storing them on AWS.
- Results Dashboard: Analyzes and displays test results, providing feedback and insights into student performance.
Impact and Future Potential: HeartMR has revolutionized the way students learn about cardiac anatomy, offering a more hands-on and engaging approach. This project not only benefits students and educators but also provides researchers with valuable data on learning in mixed reality environments.
Future Directions: The success of HeartMR opens up possibilities for further exploration in AR/VR educational applications across various fields, showcasing the potential for more interactive and immersive learning experiences.
Demo
With screen recording for review, a results dashboard, and Vuforia for AR, HeartMR offers an immersive way to learn enhancing understanding and retention for students.