Metaphors we listen with
Project Blog | Project Updates
Metaphors we listen with
Auditory brightness perception investigated through intra- and crossmodal interference
by Charalampos Saitis and Zachary Wallmark
April 30th, 2023
“It is a difficult matter to define tone quality in words; we must encroach upon the domain of sight, feeling, and even taste.”
“A sound’s timbre describes its harshness or softness, its dullness or brightness.”
Intro/ Overview
Timbre is commonly described using vision and touch terms. Auditory brightness is among the most studied “metaphors we listen with” and arguably among the most important musical cues actively shaped by performers, composers, and audio engineers. Psychoacoustically, sounds described as ”bright” vs “dull” or ”dark” typically exhibit a high vs low frequency emphasis in the spectrum. However, relatively little is known about the neurocognitive mechanisms that facilitate using a visual quality to talk about something that sounds. Our project aims to investigate the neural mechanism by which timbre activates a semantic correspondence between the auditory and visual modalities when a sound is described as bright by means of functional magnetic resonance imaging (fMRI). Is it plausible that such mechanism originates in universal mental representations common to more than one sensory modality?
Behavioural pilot study
Triangulating three different interaction paradigms, we investigated using speeded identification whether unimodal and crossmodal interference occurs when timbral brightness, as modelled by the centroid of the spectral envelope, and 1) pitch height, 2) visual brightness, 3) numerical value processing are semantically incongruent. In four online pilot experiments varying in priming, onset timing, and deadline, 58 participants were presented a baseline stimulus (a pitch, gray square, or numeral) then asked to quickly identify a target stimulus that is higher/lower, brighter/darker, or greater/less than the baseline, respectively, after being primed with a bright or dark synthetic harmonic tone. Additionally, in the pitch and visual tasks, a deceptive same-target condition was included. Each experiment involved two baseline stimuli and thus a total of four targets (two per baseline). Audio stimuli were additive harmonic complexes up to 10 kHz.
Behavioural Results
Bird’s-eye view:
We found that timbral brightness modulates the perception of pitch and visual brightness, but not numerical value.
Intramodal interference:
Semantically incongruent pitch height-timbral brightness shifts produced significantly slower choice reaction time and higher error compared to congruent pairs; timbral brightness also had a strong biasing effect in the same-target condition (i.e., people heard the same pitch as higher when the target tone was timbrally brighter than the baseline, and vice versa with darker tones).
Timbral brightness affects pitch height comparisons (Exp. 1)
RT: 𝜒2 = 67.5, p < .0001; accuracy: 𝜒2 = 199.4, p < .0001
Pitch height differences affect timbral brightness comparisons (Exp. 4)
RT: 𝜒2 = 34.8, p < .0001; accuracy: 𝜒2 = 166.7, p < .0001
Crossmodal interference:
In the visual task, incongruent pairings of gray squares and tones elicited slower choice reaction times than congruent pairings.
Visual * timbral brightness interactions in Exp. 1 & 3, respectively
RT: 𝜒2 = 4.73, p = .03; and 𝜒2 = 10.5, p = .001
Non-significant interactions in Exp. 2 & 4; no effect on response accuracy
No interference was observed in the number comparison task.
Numerical value task: null (Exp: 1–3)
We are currently following up on these results with a functional magnetic resonance imaging (fMRI) study using modified pitch-brightness and auditory-visual brightness to investigate the relevant neural mechanisms.
Our research sheds light on the multisensory nature of experiencing timbre.
