"Mind Writing" opens a new window for the exchange of ideas for paralyzed patients

Perhaps, we can call this "mind writing".

Sitting in front of a screen, without using any physical movements, you can move your "mind" to present the characters you want to write in your brain on the computer display - its "writing" speed is even comparable to the speed at which healthy peers text on smartphones.

In a report published in nature on the cover on May 13, Beijing time, researchers at Stanford University in the United States used the brain-computer interface (BCI) to allow participants to use "opinions" to write 90 characters per minute, which is more than twice as good as the previous record of using a similar "brain-computer interface" typing record, and the writing accuracy rate exceeded 99%.

"As it goes a step further, this innovation allows paralyzed patients to type quickly without having to use their hands." Krishna Shenoy, senior author of the paper and a researcher at the Howard Hughes Institute of Medicine at Stanford University, said in an interview with China Science Daily.

"Thought-driven" communication

"My flesh is as heavy as a diving bell, but my heart longs to fly freely like a butterfly..."

On December 8, 1995, a sudden catastrophe struck Jean Dominique Bobby, the editor of the French magazine who was proud of his family business, and his brainstem suffered a sudden stroke, which triggered an extremely rare and low probability of atresia syndrome. With a tenacious will and an only able to move his left eye, Bobby left behind a book called "The Diving Bell and the Butterfly."

Currently, bobby's eye-tracking keyboard technology allows paralyzed people to type 47.5 characters per minute, much lower than the able-bodied person's 115 characters per minute. And this technology is not suitable for those who have impaired eye movements, and many patients can only be isolated.

The latest research results of Shenoy and his collaborators have undoubtedly opened a new window for them to communicate with the outside world.

"When an injury or illness deprives a person of mobility, the neural activity of the brain for some behaviors (such as writing, walking, talking, etc.) still exists, and we hope to use the relevant neural activity to help paralyzed or amputated people regain some of the lost ability." Shenoy said. This led him and Jaimie Henderson, a neurosurgeon at Stanford University, co-corresponding author of the new study, to collaborate on BCI since 2005.

In a 2017 study published in eLife, they implanted BCI chips in the motor cortex of the brains of three participants with limb paralysis — each about the size of a baby's aspirin pill and with 100 electrodes on top of them, which received signals from neurons in the brain's motor cortex (an area of the outermost layer of the brain) that controlled hand movements. Participants were asked to concentrate, trying to use "opinions" related to arm movements, pointing their cursor at the character and clicking.

In the study, participants code-named T5 (who lost almost all mobility below the neck in 2007 due to spinal cord injury) set the highest record: they could click 40 characters per minute.

This approach provides a "heart window" for people who are severely paralyzed and blind. However, The idea of Shenoy and his collaborators is: "Let the idea write faster!" ”

Through further research, they found that complex expected motions that involve changing velocity and curve trajectories, like writing, can be interpreted more simply and quickly by AI equations than simple expected motions such as moving the cursor in a straight line at a steady speed.

This allowed them to shift their approach from pointing and clicking characters to writing characters. "Every character in the character table is different, so they're easy to distinguish." Frank Willett, the first author and corresponding author of the new study and a neuroscientist on Shenoy's team, explained to China Science Daily that by decoding the neural signal information on the BCI chip through the AI algorithm, it is possible to speculate on the hand movement that T5 wants to do, and quickly convert the participant's written "idea" into text on the computer screen.

Soon, new ideas doubled the speed of this "thought-driven" exchange.

Comparable to the typing speed of ordinary people's mobile phones

"If the 2017 study model is similar to 'typing', then the model of our new study is similar to 'handwriting'." Willett contrasted.

T5 was the only participant in the study. Although his arm could not move, he concentrated on trying to write out individual characters from a character table on a hypothetical grid with an imaginary pen. He repeated each character 10 times, letting the software "learn" to recognize neural signals associated with that particular character he wanted to write.

The researchers then showed T5 several sets of sentences and asked him to mentally try to "write" each sentence — without using capital characters. For example, "I was interrupted and could not remain silent" and "the army landed in thirty seconds." Finally, no matter what T5 wants to write, the AI algorithm can "write" his idea on the computer screen with a delay of only about half a second.

In the next experiment, T5 was asked to copy sentences that the algorithm had never touched. Eventually, he was able to generate 90 characters per minute, about 18 words. By comparison, able-bodied people type at a rate of 23 words per minute on a smartphone.

"This approach allows paralyzed people to output sentences at almost the same speed as healthy adults of the same age typing on smartphones." Henderson said, "Our goal is for them to regain their ability to communicate via text messages." ”

According to reports, T5's transcription error rate is about one error in every 18 to 19 characters, and its freehand writing error rate is about one error in every 11 or 12 characters. However, when the researchers used the autocorrect function (similar to a smartphone keyboard) to correct it, the error rate was significantly reduced: the error rate of transcription was less than 1%, and the error rate of freehand writing was slightly higher than 2%.

"These error rates are already quite low compared to other BCI." Shenoy said. According to reports, due to legal restrictions, the BCI currently used in this study is only used for investigative research and has not been approved for commercial use.

Jose Carmena, a neuroengineer at the University of California, Berkeley and not involved in the study, commented in an interview with China Science Daily that the technology, as well as other similar technologies, could help people with disabilities. While these findings are only preliminary, they are a major step forward in the field.

"Brain-computer interfaces turn thoughts into actions." Carmena said, "This paper is a good example: BCI deciphers the idea of writing and puts it into action." ”

The "butterfly" that lets go of the heart

Due to the wide application prospects in medical rehabilitation, old-age assistance for the disabled, aerospace and military fields, at present, domestic related research is also in full swing.

For example, the Brain-Computer Interaction Laboratory of South China University of Technology has realized the "brain-controlled input method" by collecting and analyzing the brain waves of experimenters; the University of Chinese of Hong Kong has converted brain waves into Chinese Traditional words through the BCI system, giving patients who are paralyzed and unable to speak have the opportunity to "open the window of the heart".

In addition, as early as 2016, the neural engineering team of Tianjin University cooperated with the Chinese Astronaut Research and Training Center to successfully carry out the first human space brain-computer interaction experiment, and initially explored the feasibility of idea control in space missions by obtaining a change model of the physiological characteristics of brain-machine interaction during astronauts' space flight.

However, in a news article published in Nature at the same time, Paula Rajeswaran and Amy Orsborn of the University of Washington's Department of Bioengineering said that how to keep brain-computer interfaces performing throughout their lifecycle still needs further research. "Excellent results and benefits need to be proven to be worth the costs and risks of implanting electrodes in a patient's brain."

Going forward, the Shenoy team hopes to integrate experimenting with handwritten text input into a more comprehensive system that also includes point-and-click navigation currently used on smartphones, and even experimental speech decoding. "Having these two or three modes and switching between them is something that people are born with."

"While we can write close to 20 words per minute, our rate of speech tends to be around 125 words per minute, so speech will be another exciting direction to make up for the shortcomings of handwriting." Shenoy said, "If these systems are combined, they can provide patients with more options for effective communication." ”

Next, Shenoy said, the team intends to work with deaf-mute participants, such as people with amyotrophic lateral sclerosis, a degenerative neurological disorder that leads to loss of motor and language skills. Henderson also added that the new system has the potential to help people who are paralyzed by multiple conditions. Such as people who have lost their upper limbs or language ability due to spinal cord injury.

"Bobby took a lot of effort, such as using eye movements to choose one character at a time, to write the moving and beautiful book 'Diving Bells and Butterflies.'" He said, "Imagine what he could do if there was such an interface!" (Feng Lifei Zhang Siwei)