Kubovy Perception Lab
UVA Lab


The sound is the key

Most of the studies to this point have focused on the visual aspects of the effect, but what about the sound? In this experiment we found that percussive sounds, or sounds that would naturally come from a collision (like a mallet head striking a solid surface, or a ball bouncing off of the ground), behave differently than "flat envelope" sounds which have often been used in psychology research in the past.

Below is a visual representation of these different sounds. Note how the piano note, which has what is called a "percussive" envelope, goes quickly to its peak volume and then gradually dies away. The other notes, in contrast, rise to a level volume which is sustained and then die off abruptly. This type of sound is reminiscent of a "flat" envelope sound.

We found that when percussive sounds are played along with the video their perceived duration is affected, but the perceived duration of non-percussive sounds is unaffected by the video. This explains why current theories of sensory integration claim that vision does not affect perceived auditory duration: they are all based on experiments done with flat envelope sounds! Researchers used pure tones with an abrupt onset, level sustain, and abrupt offset. These do not pair with visual events in the same way as percussive sounds do, as this experiment illustrates.

Non-percussive timbres (clarinet and voice)
Percussive timbres (piano and marimba)

The videos have a significant influence on perceived duration ratings of percussive sounds, but not on non-percussive sounds.

As you can see, the percussive and non-percussive sounds behave quite differently. The consequences of this finding are quite important and interesting-- check back for an updated section explaining why.

 

Back to Sensory Integraton