How can we make virtual reality tactile
Augmented and virtual reality is going to transform healthcare. But in a world where so much is relayed through touch, can we truly create an immersive healthcare experience.
Augmented and virtual reality show tremendous promise in healthcare. However, there is one glaring issue with this new technology, and that is its lack of tactility.
A key part of being a doctor is your examination skills. You need to be able to palpate for a mass, percuss for subtle changes in sound, and feel for increases in size as in measuring chest expansion. Even more so in surgery, where the slightest movement depends on the resistance of particular tissues. In the real world, these are not complex tactile things to do (relatively). But in a virtual world, recreating these interactions becomes incredibly difficult. But not impossible.
Methods of creating tactility
There are multiple ways in which we could solve this issue, I think it is useful to split them into 3 modalities:
- Object-driven
- Peripheral Sensation
- Central simulation
Below I will discuss the theory behind these and current examples within each.
Object-Driven
Object driven is exactly that, using real world objects and mapping the VR object to it. This is by far the easiest of approaches to generate tactility in a VR world. Some may think this the same as augmented reality, but it is in fact the opposite, which has lead some to dub this ‘augmented virtuality’.
By tracking real world objects, e.g. a ball, and mapping it to a virtual object of the same size, users interact with the real world object, but feel as if they are interacting with whatever object is projected into their virtual world. Far from the future, companies are already using this form of virtual reality to fully immerse users in a virtual world. An unlikely suspect is Disney, who have experimented with mapping a ball into a virtual reality landscape to play a game of catch in VR.
It is likely that this form of tactility in virtual reality will be the earliest form of fully immersive VR. While still technically complex, it is the simplest form of relaying tactile information, as there is no need to simulate the touch, only map a virtual representation onto a real world object.
As I’m sure you can see, this approach has limitations. Having to create real world objects for all items that you can interact with in a virtual world doesn’t feel very ‘virtual’ at all, and in fact rather defeats the point. However, given the absence of any other method, it shows promise with small VR worlds that want to build fully immersive experiences.
Peripheral Sensation
As oppose to object-driven, we begin to stray into the science fiction with the modality of peripheral sensation. First of all, what I mean by peripheral sensation is genuine sensation, without bypassing the users peripheral nervous system, but without the need to generate real world objects.
This concept is similar to the ‘Haptic suit’ in the film Ready Player One. The haptic suit provides feedback to the user from experiences within a virtual world. An initial attempt at this suit is the Teslasuit (nothing to do with the electric car company).
The Teslasuit is able to recreate the sensations of touch and even feelings of physical exertion and temperature. It does this using vibration and thermal components built into the suit. It also tracks important biometrics, including EEG and Galvanic Skin Response sensors making it able to detect the level of stress the users body is under. This comes in handy if you want bidirectional sensory behaviour, i.e. as well as the suit relaying sensory information to the user, the virtual world can adapt to biometric data taken from the user.
This suit is a great start in creating peripheral sensation for virtual worlds, however, a key issue it doesn’t address is interacting with objects within that space, i.e. the tossing of a ball we discussed earlier. Although you could feel the touch of the ball, you would not feel the pull of the ball due to gravity, i.e. its weight. Currently there is no suit capable of doing this, however, that does not mean this is not theoretically possible.
One way I imagine this could be done is through the use of a smart material with an adaptable rigidity depending on an electric current. At rest (with no electricity passing through it), the material would be flexible, but if pressing against an immovable object, the material could become rigid restricting the users mobility, and therefore simulating an object, without the need for a real world surrogate to the virtual object. This type of material could also simulate moveable objects by slightly increasing its rigidity to simulate the weight of the object, but not so much as it completely restricted the users movement.
As far as I’m aware, no such material exists with this behaviour (apart from muscle of course). But with such a material it would theoretically be possible to simulate virtual reality tactility without any need for real-world objects, transforming our ability to produce large immersive VR worlds.
Central Simulation
I must stress, as I begin to discuss the modality of central simulation, we are truly stepping into the realms of science fiction, though I will try and keep it grounded within scientific fact.
First, a bit of science. The human nervous system is split into the peripheral and central nervous system. The peripheral nervous system is all the the nerves going to and from your peripheries to your central nervous system. Your central nervous system is your brain and spinal cord.
Central simulation describes the process of bypassing the patients peripheral nervous system entirely, manipulating the central nervous system directly to simulate experiences to the patient.
Within the brain there lies the primary somatosensory cortex. This is the area of the brain that, although a massive oversimplification, processes sensory stimuli. This area is helpfully mapped out into the sensory homunculus, meaning a portion within it relates to the hand, another section the face, and so on. These sections go in order, from your feet going up your body to your head. Useful if you would like to recreate a sensation in only one location.
Therefore, if we could activate this area of our brain in the appropriate way and with enough accuracy, we could simulate senses such as touch and temperature without need for these stimuli to be present in our environment. The advantages to this modality of sensory simulation are tremendous. We would be able to build entire virtual worlds, with complete sensory integration. Perhaps disturbingly, it would be difficult to tell if we were in a real world or not.
Although this seems a distant prospect, companies are already attempting to build this form brain-machine interaction. Notably, Elon Musks start-up, NeuroLink, is attempting to design ultra-high bandwidth brain-machine interfaces to allow a connection between humans and computers. It is unclear if they have made much progress with this, though most feel it is unlikely given the complexity to the challenge they are attempting to overcome.
An issue with this method arises with how you restrict a users movement. For example, imagine there is a wall in your VR world. You push against it and a sensation of a wall is simulated within you somatosensory cortex — so you can feel the wall. But, there is no wall to actually push against, and so nothing to stop you moving ‘through’ the wall, i.e. nothing to stop your movement. This would create a tremendous degree of sensory confusion, the effects of which we do not know.
Summary
Virtual and augmented reality is likely to become a tremendous tool within healthcare and medicine within the coming years, however, it is hindered from creating truly immersive experiences due to its lack of sensory simulation — and while some senses are easy to fool, tactility is not one of them.
If we are to use these tools in education and clinical settings, especially in areas such as surgery, where touch is integral, then novel approaches to recreating these experiences will need to be developed. Above are examples as well as some of my ideas as to how this could be done, but these are far from exhaustive, so please add any ideas you have in the comments section below.
And if you liked this article remember to follow Doctors in Tech and like us on Twitter and Facebook.
