Scientists put visions of letters in blind people's brains

Stimulating the brain in specific ways can generate mental images of simple shapes

Image:

NSF via YouTube

Surrounded by the buzz of medical equipment, a blind man raises his hand to a touch screen. Pop! A vision of the letter "N," placed in his brain, flashes through his mind. He traces his finger across the screen, replicating the vision with perfect form.

It sounds like science fiction. But in a recent study at Baylor College of Medicine, researchers made the blind see. A team led by neurosurgeon Daniel Yoshor "drew" letters of the alphabet on blind people's brains by giving them specific patterns of electrical zaps. These patterns caused the participants to "see" the letters in their mind's eye. The results could improve medical devices for people who have experienced other types of sensory or motor loss.

A man traces a shape, stimulated in his mind, on a computer screen with his finger

A participant in the study is shown drawing a letter that was stimulated in his brain

Screenshot via Daniel Yoshor on YouTube

The researchers accomplished this by giving patterns of small electrical stimulations to the visual cortex. The visual cortex is one of the hubs in the brain that responds to what we see. This region contains a spatial map of our field of view, meaning particular sets of cells respond to visual information coming from particular locations in our line of sight. Turn on a light on the left side of your field of view, and one set of cells will respond by shooting off an electrical signal. Turn on a light on the right side, and a different set of cells will respond.

Yoshor's team took advantage of this map in a clever way. Because the cells in the visual cortex respond to patterns of light in space, the scientists could reverse the process — give a tiny electrical zap to a particular group of cells and cause someone to perceive a spot of light at a specific location. They performed brain surgery on blind adults to implant a small electrical device with several points of contact to the visual cortex. Each point could be activated individually or in combination with others to stimulate the brain in precise patterns. By carefully controlling the combinations of activated areas, the researchers could cause someone to "see" a specific shape, such as a letter of the alphabet.

But there was one problem. Other researchers had tried similar experiments, but they hadn't been successful in creating a detailed image in someone's mind. These previous experiments stimulated discrete patches of cells, like a shape drawn with dotted lines. But instead of seeing a continuous shape, people just saw several blobs of light. In order to create a detailed mental image, Yoshor's team had to improve on existing techniques for stimulating the brain.

They realized that they could borrow a technique that's common in cochlear implants and auditory therapy to improve the way they stimulated the brain's visual center. This new technique allowed them to replace their dotted sketch with a single continuous line to "draw" shapes such as letters on the brain. They reasoned that the brain might be able to make more sense of a stimulation pattern that is drawn continuously, since the pattern would more closely match the appearance of most letters in daily life.

To achieve this continuous drawing in a person's brain, the researchers gradually changed the level of activation of each point of contact. Instead of turning each point just on or off, they raised or lowered the dial on each one to create a wide range of activity in the corresponding brain cells. When they gradually increased the activation of one point of contact while gradually decreasing the activation of another, they caused people to "see" a continuous line of light corresponding to the space between the two active points of contact in the brain. Participants described the image "like a line being drawn" in their mind's eye.

Now that the researchers had improved their technique for stimulating the brain, they were able to test whether complex activation patterns could cause people to perceive shapes like letters. They stimulated groups of several points in the brain, adjusting the dial at each point to draw lines in spatial arrangements that matched the form of real letters. They asked people to replicate the shapes they saw on a touch screen, and also to respond aloud to indicate which letters they perceived.

prosthetic arm

The results from this study may help people better control their prosthetics

Photo by ThisisEngineering RAEng on Unsplash

The participants were consistently able to reproduce and name letters that matched the stimulation pattern drawn on their brains. When asked to name letters in rapid succession, they accurately named nearly 90 percent at a rate of 85 shapes per minute. In other words, the new stimulation technique successfully generated clearly shaped letters in people's minds.

But the technique isn't perfect. The activation patterns aren't precise enough to allow people to tell the difference between similar shapes, like the letters "C" and "O". And the participants in this experiment became blind as adults, meaning they already knew what the letters looked like. It's likely that it would have been more difficult to replicate the experiment's results in someone who had been born blind and had never seen letters or other shapes.

Nevertheless, this experiment is a promising step forward in scientists' ongoing effort to create a device that could restore vision to those who have lost it. With further improvements in the technology, scientists might be able to stimulate the brain with a finer resolution and generate more detailed mental images. The results could have implications for other types of sensory function, too. Finer stimulation methods could improve pitch perception in users of cochlear implants, where a version of this technique has already been tried, and could help restore a detailed sense of touch in cases of nerve damage.

Besides sensory systems, the results might also improve medical devices that help with motor control. For example, prosthetic limbs rely on technology that reads activity in the motor regions of the brain. Scientists could use their new knowledge about stimulating the brain to read and interpret the brain activity that controls a prosthetic, allowing people to control their new limbs in more precise ways.

It's clear that the results uncovered by Yoshor's team are a crucial development in the ongoing quest to create a visual aid for the blind. In the future, the technology might allow blind people to see the complex objects that surround them in real time.

Peer Commentary

Feedback and follow-up from other members of our community

Thanks for writing this article, Meredith, it highlights exactly the kind of spectacular research that made me want to be a scientist in the first place!

I especially liked how you used the dotted line analogy to explain how this study made improvements to the resolution of techniques that were already available. It really makes me think of what could be next: shading? Or even adding color?

Of course that would require stimulating more/different parts of the brain, but perhaps as their method gets less invasive!

Wow! I thought this was a great choice of topic for an article. I remember reading this paper when it came out, being very pleased that artificial vision is continuing to progress. I think an interesting take-away for readers might be the retinotopy of the visual cortex, which you alluded to when describing the spatial map. I think a reader might find it interesting to know that not only do different visual cortex cells respond to different places of the visual field, but they are arranged so that the cells next to each other in the visual cortex also correspond to spots in the visual field that are next to each other. This maintenance of retinotopy (as you are already an expert in!) is what is so beautiful and efficient about the visual system, and is what allows this kind of technology (though novel and in its first stages) to work.

This is such an interesting topic, and a great explanation. The first thing that came to mind for me was an experiment that I participated in on haptic (the science of touch), where they found that participants were better able to distinguish shapes drawn on their hands when they were drawn as a line rather than presented as a single shape. It seems like this is a common feature of our sensory systems. I wonder how much this is related to our tendency for exploratory behaviors (such as eye saccades).

I do think an important point to be made surrounding this, is the ethics. In order for a technique like this to work, the participants had to be implanted with electrodes. This is most definitely not risk-free (see e.g. this article). Of course, if this technique can help blind people to see again, that is great, but not everybody might want to have, or have access to, such a procedure. Ultimately, it should not replace other means of accessibility.

Fascinating article. At the end, you mention that this technology may “allow blind people to see complex objects that surround them in real time.” What do you think are the major barriers to go from recognizing letters to seeing arbitrary objects? How quickly can an object be drawn and at what resolution? Will software and hardware advances be needed to distill a scene into a drawing, or do you think existing off-the-shelf components can be used?

Meredith Schmehl

Hi Vidya, these are great questions! Some of the major barriers include translating the techniques developed here for drawing 2D objects (letters) to later draw 3D images of everyday objects, as well as increasing the resolution of the perceived images. And as you mentioned, the speed of the drawing is another barrier - although the scientists in this experiment were able to draw each image in less than one second, the technique isn’t fast enough to convey complex information about a visual scene, such as objects in motion. To improve in these areas, scientists will need to develop finer stimulation methods, and participants may need to be trained to recognize particular stimulation patterns.

Vidya Ganapati

Thanks!  One could even imagine something like “Google Maps” for the brain where directions are given but fine-grained scenes are not. Of course, that would require accuracy guarantees and other considerations. 

Great article! As Dori mentioned, I’m curious about the ethical component of this study. Is the risk of implanting electrodes worth the potential benefits of this technology? I see that participants had intractable epilepsy and therefore needed  these electrodes to be implanted anyway. This therefore presented a great opportunity to understand the mechanics of cortical stimulation and investigate this new technique, which may inform more non-invasive approaches in future. I wonder whether similar results could be achieved using haptic stimulation on the skin surface.

It’s also interesting that the researchers chose to stimulate letters from the Latin alphabet. Their discussion mentions previous studies which used visual braille. I’d be curious to see how this new technique could work with braille, and whether this might address a limitation you brought up, making the technology translatable to users with congenital  blindness who have never seen letters, but may have learned braille. That said, I recognize that letters are a gateway to more complex visual scenes, and suspect there would be few advantages to stimulating braille in the brain when braille readers’ sense of touch is already so effective!

Meredith Schmehl

Thanks for your comments, Danielle! You’re correct that the stimulation technique is quite invasive. I’d love to see similar results using stimulation of the scalp instead, but currently the ability to activate specific neurons from the skin surface is not precise enough to replicate stimulating the brain itself.

The “visual braille” component is definitely an interesting idea that the authors present in the paper. Since it’s difficult to replicate shapes perfectly with their stimulation technique, they could instead train people to recognize particular patterns as having particular  meanings. This could greatly increase the number of shapes or objects that people can “see,” potentially allowing people to know more about what’s in their environment. We’ll have to see whether this can be done  using the current stimulation techniques, or if further advancements are needed.

What a fascinating study. Your description of the complicated methods is written in such a way that anyone could understand, no matter their scientific knowledge or  background.

Do you think that, for those willing to undergo the surgery and  implantation, a computer program will be able to image their  surroundings and ‘draw’ in real-time? Or will it require a scientist or carer to draw for them?

Meredith Schmehl

Great question. I’m not sure that computer vision technologies are currently able to “see” an environment in enough detail to translate to an accurate drawing in the brain, but that would be interesting to try in the future! I imagine that method would eventually be a more  sustainable option than a human artist, but first we’d need more advancements both in computer vision and in the brain stimulation technique.