Capturing movements for a more immersive experience

As mentioned in a previous article on motion capture and finger tracking, motion capture enables the digitization of the gestures performed by the learner during the simulation. Capturing gestures in virtual reality offers several advantages, including a more immersive experience. Indeed, users can use their hands and bodies to interact with the virtual world. They can move freely within the virtual environment, without being limited by controllers. Interactions with virtual objects are done in a more natural way, with gestures similar to those in the real world. Motion capture allows for tracking the movements of hands and fingers with great precision, allowing for finesse and subtlety. It is therefore understood why many stakeholders are interested in it and are driving its progress.

Once cumbersome to implement or limited to studios equipped with high-end equipment, motion capture is becoming more accessible and lightweight thanks to new solutions available.

Hand gestures

After discovering Senseglove‘s haptic gloves at the 2022 exhibition, our attention has turned to Manus Meta’s Metagloves this year.

Manus Meta Gloves – Quantum Metagloves (5,999 €)

The feedback from our testers :

Practical, the gloves can be easily calibrated to fit the user’s hand size. The capture of finger skeleton movements (finger tracking) appeared to be highly accurate. However, the test was conducted with a single glove, so it was not possible to experience a VR immersion demonstration involving both hands. Thus, some doubts remain regarding the accuracy of 3D hand tracking in terms of the spatial positioning of one hand relative to the other or to the virtual reality headset.

The technology, however, is mature and now has a sufficient level of validation to be reliably integrated into simulators. It also interfaces seamlessly with Unity through a plugin.

Ultimately, it is suitable for dedicated uses where the analysis of delicate finger gestures is required, such as fine craftsmanship, surgery, tactile industrial quality control, and more. However, the ease of use could certainly be further improved, especially in terms of equipment setup time, and the cost remains prohibitive for widespread adoption.

For use cases that require basic finger tracking within the field of view and do not require millimeter-level precision, the finger tracking provided directly from the cameras of the headsets can currently be sufficient. This will also allow for a simpler setup to be maintained.

Analysis of body movements in virtual reality

Traditional optical-electronic motion capture systems (such as Vicon, Qualisys, …) are still considered the gold standard in terms of accuracy. However, they are still too cumbersome and expensive to be implemented in an immersive learning setup.

Therefore, suits equipped with inertial sensors have gradually emerged as a lighter solution. For instance, the manufacturer Movella (with its well-known X-Sens solution) was once again represented at Laval this year. However, their solution still requires a significant financial investment and requires a certain level of expertise to be implemented properly.

However, this sector has not been spared by the AI revolution. The startup area of the exhibition featured move.ai, a company showcasing its technologies capable of measuring full-body movements using just video cameras. And this can be done even using a smartphone camera, without the need for additional sensors or markers.

This solution, extremely easy to deploy, may not offer high precision right away, but it is already sufficient for analyzing the body as a whole and its posture. For example, it can capture the movements and gestures of the learner in the virtual environment and use the data in real-time to provide feedback and guidance. It can helps learners to improve their performance by allowing them to see their movements and compare them to correct movement patterns. Furthermore, it can allow for a detailed analysis of the learner’s performance progression, which can be useful for evaluation purposes. Motion capture helps make learning more interactive, engaging, and effective.

One example of use immediately comes to mind: raising awareness about musculoskeletal disorders (MSDs) risks during manual handling operations.

Measuring the emotions of a learner in immersion

Virtual reality immersion is increasingly being used in the field of training and education. However, it can be important to measure the emotions of the learner to improve the effectiveness of this learning method. For example, determining if the learner is able to perform a task in a degraded situation (stress management, reactions to danger, etc.). With this in mind, virtual reality tools have been developed to measure the physiological and emotional reactions of the learner during their VR experience.

The startup Kaptics was present at the Laval Virtual startup hall. The company presented a device for evaluating the user’s emotions during immersion. It is based on a technology that integrates biosensors into a traditional VR headset (the setup currently requires the use of an adapted strap). The solution measures EOG (electrooculogram) signals, EEG (electroencephalogram) signals, and facial EMG (electromyography) to track eye and brain activity as well as facial expressions.

Exploiting this information involves a colossal amount of signal processing work, including sampling, filtering, denoising, and more. Thus, there may have been concerns about a highly complex usage limited to research laboratory settings. This is not the case, as the company has skillfully extracted the relevant information and limited the output to 4 measurements:

  • the level of attention (focus, especially thanks to EOG which can act as an eye tracker),
  • the level of relaxation,
  • the emotional state
  • heart rate (the last three measurements help estimate the learner’s stress level).

It is easy to imagine how this approach can be integrated into our simulations. For example, it can provide the learner or their trainer with valuable objective and quantitative data. Measuring the stress level of a learner during a simulation involving the handling of hazardous materials or operating a large construction vehicle, for example, can be valuable.

The solution, currently being commercialized, is already available for experimenting with proof of concepts.

For more information on the importance of emotion in learner training, you can read our article “Add Emotion to Your Pedagogy.”