To give you a natural control AR glasses, Facebook AR explore interactively from a variety of directions, such as: a camera, somatosensory gloves to track gestures, or track fingers with EMG technology, and so on.

In fact, in addition to natural interaction, the tactile feedback of the fingers can also help increase the immersion of AR. So, how to combine tactile and gesture interaction naturally? To explore this problem, Facebook researchers have developed a somatosensory enhancement system that changes vision, sound, and touch: Tactile Echoes.

Tactile Echoes co-designer Anzu Kawazoe (University of California PhD candidate) said: Tactile Echoes may be the first somatosensory solution that combines multiple analog feedback such as AR projection and gesture recognition. It has 10 adjustable parameters and supports simulating diverse multi-sensory feedback.

It is reported that Tactile Echoes is also called a “gesture-based multi-sensory enhancement system” by researchers. It is characterized by a wearable design combined with a projection-based AR interface. Echoes is based on AR projection, gesture tracking is through the camera, and somatosensory feedback through hand refers to a piezoelectric sensor and actuator side to achieve.

The advantage of using the AR projection interface is that the experiencer’s hands can directly touch the desktop and get feedback, without control buttons or touch screens for input, so the experience is more natural, and it can integrate visual, somatosensory, and auditory multiple sensory feedback. In terms of application scenarios, it can be used for games, entertainment, scientific research, etc.

In detail, the entire system includes: piezoelectric sensors, actuators, sound cards, PCs, speakers and audio amplifiers. After the piezoelectric sensor recognizes the finger vibration signal, it will be transmitted to the PC for processing, and output to the actuator to generate vibration feedback to the finger, and at the same time the speaker will also send out audio feedback. And AR projection provides visual UI.

In fact, in daily life, when you tap on the desktop with your fingers, you can usually hear the click. What the Tactile Echoes solution wants to know is whether you can use hollow echoes or roaring thunder to replace the clicks. .

In other words, the camera recognizes the intensity of the vibration when the finger is in contact with the plane, which will affect your interactive experience with the AR interface. The somatosensory and audio feedback produced by Tactile Echoes is related to the force with which the finger touches the surface. The greater the force, the stronger the feedback. When you touch something, you may hear a soft echo. When you poke something hard, You may hear a pop echo. And when your fingers touch different interfaces, the feedback you get is also different.

For example, when you tap the AR interface with your hand for a short time, you will get a discontinuous echo. If you slide your finger on the AR interface, it will produce a continuous echo. In addition, different somatosensory feedbacks are rendered according to the surface or position of the touch screen.

Kawazoe said: Our inspiration is to use Tactile Echoes to enhance the touch of ordinary surfaces such as wooden tables, adding dynamic somatosensory feedback and sound effects to respond to touch.

Facebook said: Tactile Echoes is a wearable tactile feedback system, which is characterized by capturing the vibration generated by a finger touching a table and other surfaces, and processing the vibration data in real time, transforming it into “echo feedback”, just like converting tactile vibration into sound Vibration, which produces a continuous feedback process of echo.

In the experiment, the researchers created a 2D AR music synthesis interface, which seems to be sufficiently interactive. Although it is only a virtual AR interface, it can also provide natural feedback of touch, vibration, and audio at the same time.

Researchers used VR/AR headsets and interactive AR projections to show visual effects. The experiment is divided into two types. One is to investigate the experiencers’ cognition of different sound effects and somatosensory feedback. In the second experiment, the experiencers need to complete an interactive AR game combined with the Tactile Echoes solution. Experiencers said: Audio and somatosensory feedback have effectively improved the responsiveness of game interaction, and the sense of immersion and presence has also been strengthened.

In the first experiment, the experiencers went through a total of 35 tests, each time they needed to use verbs or adjectives to describe their feelings. Researchers investigated the experiencers’ views on the combination of echo feedback and tactile sensation, and through some tactile games, let the experiencers compare the experience of Tactile Echoes and the difference between traditional somatosensory games.

Researchers have also discovered some interesting phenomena related to perception, such as delayed somatosensory feedback and perceptual masking, which means that a certain stimulus will affect people’s perception of the intensity of another stimulus. In the future, researchers will continue to conduct experiments in related directions and use the results to optimize the entire Tactile Echoes system.

In addition, Facebook plans to cooperate with companies to commercialize Tactile Echoes for use in desktop games, music performance, education and other scenarios.

refer to:

https://spectrum.ieee.org/tech-talk/consumer-electronics/gadgets/new-ar-system-alters-sight-sound-and-touch