Synopsis:
FlexVR Wellness is an ecosystem developed to facilitate electro-stimulation therapy over XR environments, aiming to reduce patient stress. The system allows therapists to use AR headsets to communicate with and remotely control therapy sessions in calming VR environments.
This system was created in 60 hours at MIT Reality Hack and won the award "Hardware: Creative Inputs/Outputs." It was created by Cade Gilbert, Shane Mckelvey, James Higdon, Peter He, Ty Lindell, and mentored by Rina Kim.
Understanding The Problem:
When we started working on this project, we had a question we wanted to answer: How can we make a patient's experience, in any field, more comfortable, without making the professional's job harder? We understood that VR scenes could be used to visually stimulate patients in a calmer environment while a professional was at work and decided to build on that idea. For our demo, we used a programmable device called a StimDuino, an open-source Arduino-based board to allow us to fine control electrodes using code, as well as integrating it with a suite of XR devices using OSC and Unity. With this device, we were able to create a demo for an asymmetrical end-to-end therapy experience. On one end is the patient, who is in a calming environment in VR, and on the other end is the therapist in a workspace environment using AR glasses. Using OpenXR, not only can these two end users hear each other, but they can also see each other in 3D space, allowing for a more personable experience.
Understanding - The Patients Experience:
The patient's experience consists of two major parts: the VR scene and the Electro-Stimulation device (E-Stim Device):
The VR Scene: A scene built in Unity 2022 with distracting colors and visuals, allowing for the patient to be immersed and not focused on the fear that can come from electro-stimulation therapy.
The Electro-Stimulation Device (E-Stim Device): A small form factor device that can be clipped onto the belt or pants of the patient, with two electrodes running to set positions on the user's arm (the electrodes can be placed anywhere, however, for the demo, we used a position on the user's arm).
The E-Stim Device connects to the PC for this demo, either via a Bluetooth connection or wired (for the demo, it was wired). Once connected, we use OSC to allow access to the custom control code in Unity. The patient's scene is fairly simple, a calming environment for the user to sit in, as well as a 3D avatar of their therapist that is tracked to the therapist client, tracking the therapist's head and hands into 3D space to allow for a more personable experience.
Understanding - The Therapist Experience:
The therapist's experience consists of two major parts: the Lenovo Think Reality A3, running the Snapdragon Spaces platform using Dual Render Fusion, and the workplace scene:
Workplace Scene: The workplace scene is built in Unity 2021; for the demo, it was relatively basic with patient info, as well as a 3d overlay of where the electrodes needed to go on the patient, and the ability to import a heartrate monitor. While that was the demo, there is plenty of information that can be transferred to this scene. In this scene is also a 3d avatar of the patient, allowing for the therapist and the patient to be present with each other in 3d space.
Snapdragon Spaces Device: With this device, we are taking advantage of a few key features: augmented reality, hand tracking, and Dual Render Fusion. The use of augmented reality allows for easy usage in a workplace environment. Most devices coming out right now in XR are "Spatial Computing" devices that replace your workstation with a headset. Using a pair of augmented reality glasses allows the user to not only use their current workspace such as one with a computer that they are used to but add virtual overlays and still be within 3D space with their client. Hand tracking also adds to this as an easy zero-friction control environment. Using the hand tracking, you can rearrange your workspace environment without needing to connect any form of controller.
On top of this, we are using a new feature from Snapdragon called Dual Render Fusion. This allows for the AR glasses to be connected to a phone but show a different screen on the phone compared to the AR glasses. Using this feature, the glasses show all the useful information, while on the mobile phone, you can have all of the controls for the E-Stim device on the patient's body, meaning the therapist can fine control the E-Stim device on the patient's body from anywhere in the world while still being completely present with the patient.
This demo, while basic, shows a functional end-to-end system for both therapists and patients. A patient can be in a consumer device that is easily purchased or loanable with insurance, while the therapist can be in their own office controlling the entire session as if they were with the patient.
Looking to the future:
While this was a simple demo, there are many key areas where it can be improved. For example, we can cut out the need for a computer entirely for the patient by making a specialized E-Stim controller that connects to the patient's mobile phone via Bluetooth and then connects wirelessly over the internet to the therapist's control system. This specialized device can be made much smaller and cheaper due to the lack of need to control it directly on the device.
For the therapist, we can integrate more systems to help them in their workflow, such as more patient data to the AR scene, finer controls on the mobile app, and the ability to pass 3D models between the therapist and the patient to help the therapist explain info to the patient.
One other system we thought of but did not have the time during the event was to create a system where we could use AI to 3D model environments based on patient information prior to the appointment. This environment can be generated and imported to provide a more personal experience as to calm the user to help them feel more familiar with their environment.
While Reality Hack is over for this year, the team is still interested in continuing development of this ecosystem, if you wish to contact us, please reach out at:
flexvr.wellness@gmail.com
Or connect with me on LinkedIn:
https://www.linkedin.com/in/cade-gilbert/