Need to Listen Carefully? Look Away! cover

Need to Listen Carefully? Look Away!

By


A dish clatters to the floor, and you spin around to view the damage. A friend calls out from beyond your line of sight, and you turn toward the sound. We’re instinctively aware that looking at the source of a sound makes it easier to understand—except when your eyes trick your brain into hearing things.
In a phenomenon known as the McGurk illusion, the syllables you hear sound different if you simultaneously watch a person’s mouth moving in the shape of another syllable.
Seeing is believing for this!


Rating: 4 out of 5 stars on 1 review




NoteStream NoteStream

NoteStreams are readable online but they’re even better in the free App!

The NoteStream™ app is for learning about things that interest you: from music to history, to classic literature or cocktails. NoteStreams are truly easy to read on your smartphone—so you can learn more about the world around you and start a fresh conversation.

For a list of all authors on NoteStream, click here.




Read the NoteStream below, or download the app and read it on the go!

Save to App


Need to Listen Carefully? Look Away!

The McGurk Illusion

A dish clatters to the floor, and you spin around to view the damage.

A friend calls out from beyond your line of sight, and you turn toward the sound.

We’re instinctively aware that looking at the source of a sound makes it easier to understand—except when your eyes trick your brain into hearing things.

In a phenomenon known as the McGurk illusion, the syllables you hear sound different if you simultaneously watch a person’s mouth moving in the shape of another syllable.

His Master's Voice

His Master's Voice

In England, artist Francis Barraud (1856-1924) painted his brother's dog Nipper listening to the horn of an early phonograph during the winter of 1898.

Victor Talking Machine Company began using the symbol in 1900, and Nipper joined the RCA family in 1929.

Public Domain Image

Illusion

Wikipedia describes the McGurk effect as a perceptual phenomenon that demonstrates an interaction between hearing and vision in speech perception.

The illusion occurs when the auditory component of one sound is paired with the visual component of another sound, leading to the perception of a third sound. The visual information a person gets from seeing a person speak changes the way they hear the sound. If a person is getting poor quality auditory information but good quality visual information, they may be more likely to experience the McGurk effect.

(CC BY-SA 3.0)

The Illusion

Being aware of this audio-visual trick doesn’t stop your brain from falling for it over and over again, though watching subtitled movies can help a little.

Recently published PLOS ONE research shows that the illusion is caused by visual signals reaching the auditory cortex in the brain faster than the sounds are processed by your ears. Researchers analyzed brain signals in the auditory cortex, the part of the brain that processes sound, when volunteers were given a combination of videos and sounds to watch and hear.

The Auditory Cortex

The Auditory Cortex

Photo Courtesy Mikael Häggström

(CC BY-SA 3.0)

The Sounds

The sound clips were of the syllables “ba,” “ga,” “va,” and “tha,” but physical mouth movements in the concurrent video weren’t always the same.

In some tests, movements in the video matched the spoken sound perfectly, but in others, the sounds were either completely mismatched, like watching a poorly dubbed movie, or just slightly mismatched. Listeners had no trouble identifying the sound they heard in the extreme case of an absolute mismatch, such as a video of “ba” paired with the sound of “tha”, and they did just as well when sound and video lined up perfectly.

However, when the mismatch was less obvious, such as “ba” with “va,” participants “heard” what they saw (va), and not what was played for them (ba).

Subtle Mismatch Effect

When sounds and videos were perfectly matched or mismatched, participants’ brain activity corresponded to the auditory signals.

But, when the mismatch wasn’t as obvious, activity in the auditory cortex increased in response to what participants saw, rather than what they heard. More simply, their brains ‘heard’ what they saw, not the sound that was played.

Understanding why the McGurk illusion occurs in the brain isn’t likely to change how we experience the effect (no really, see for yourself on the next Note), but the results take us a step closer to learning how we really hear voices in our heads.

Try The McGurk Effect!

Horizon: Is Seeing Believing? - BBC Two

Video

Citation

Smith E, Duede S, Hanrahan S, Davis T, House P, et al. (2013) Seeing Is Believing: Neural Representations of Visual Stimuli in Human Auditory Cortex Correlate with Illusory Auditory Perceptions. PLoS ONE 8(9): e73148. doi:10.1371/journal.pone.0073148

Originally posted to PLOS One, (CC BY 2.5)