Visualizing Sound: Effects of Pitch Height and Tonality on Luminance Matching

By Naomi F. Maxwell
2013, Vol. 5 No. 10 | pg. 1/2 |


Synesthesia, a perceptual condition in which two or more senses intermix, has elicited research on cross-modal perception in general. Previous studies have found similar patterns of matching color to sound among synesthetes and non-synesthetes. This study tested a sample of non-synesthetes for consistent patterns of luminance matching in response to triads varying in pitch height (high versus low) and tonality (major versus minor). A repeated-measures ANOVA revealed significant main effects for both variables, which suggests that non-synesthetic populations are able to make consistent associations between audio and visual attributes in a fashion similar to that of sound-to-color synesthetes. These findings support the possibility that some of the neural mechanisms underlying sound-to-color synesthesia may be shared with normal populations.

Synesthesia is a perceptual condition which challenges the tradition notion of five completely separate senses. Synesthetes experience a blending of the senses; they have additional sensory perceptions in one modality in response to input from another. Although rare, synesthesia can occur between any two or more of the five senses, and research has verified the unusual reports of synesthetes by testing their perceptions and finding them to be unlearned, involuntary, and very consistent over time (Ward, Huckstep, & Tsakanikos 2006).

One form of synesthesia involves the involuntary perception of specific colors in response to sound, especially music. This type of synesthesia, called “colored hearing” or sound-to-color synesthesia, has sparked the interest of researchers who examine emotional correlates of music and color, as well as those interested in cross-modal, particularly audio-visual, associations made by people who lack synesthesia. Comparisons between some attributes of synesthetes’ involuntary color perceptions with audio-visual associations of non-synesthetes has fueled one theory of synesthesia, which claims that this highly unusual condition has roots in normal cross-modal mechanisms common to all human beings (Ward, Huckstep, & Tsakanikos 2006). This study examines potential cross-modal associations made between one visual attribute, luminance, with two different auditory attributes, pitch height and tonality.

Luminance and Cross-Modality

Luminance (the relative lightness or darkness of an object) is relevant to comparisons between synesthesia and normal cross-modal perception because of its associations with overarching emotional and metaphorical concepts. In a 2007 study, Meier, Robinson, Crawford, and Ahlvers found that subjects perceived squares as brighter when paired with positive words, and darker when paired with negative words. This finding suggests that the idea of congruency, or matching the attributes of distinct concepts with each other, can influence individual aspects of perception. Along similar lines, Marks (1982) conducted a study in which participants adjusted the brightness of a light and the loudness of a tone to match given lines of . The subjects adjusted the respective levels of each mode (low, medium, or high brightness or loudness) in almost perfect correlation, proportionate on a line graph. Since subjects in both of these studies seemed to match luminance, a strictly visual attribute by definition, with external, non-visual attributes, we can reasonably conclude that luminance may represent a broader concept, such as intensity, which lends itself to cross-modal mapping.

Hubbard (2010) compared previous empirical findings related to auditory imagery, and suggested that since auditory imagery shares many semantic characteristics with acoustic stimuli, humans’ interpretation of both may be based on shared qualities, such as intensity. Based on this theory, and the findings of Meier et al. and Marks, luminance seems to aptly represent visual intensity, as demonstrated by subjects’ ability to effectively associate it with non-visual concepts.

Pitch Height and Luminance

Research has examined correspondences between pitch height and luminance using several methodologies. In a series of three experiments conducted by Ward, Huckstep, and Tsakanikos in 2006, a group of sound-to-color synesthetes and a non-synesthete control group matched colors with pitches, using a custom color picker on a computer. Although the synesthetes demonstrated much more consistency in their color choices, both groups chose lighter colors for higher pitches, and darker colors for lower pitches. A study by Miller, Boteler, and Larcombe (2003) observed a similar response pattern in a case study of one synesthete, finding that lower-pitched tones elicted larger and darker visualizations than high-pitched tones.

In a 2012 study, Walker found that pairing a light or dark grey background with either low-pitched or high-pitched sounds affected how quickly non-synesthetes correctly completed a word classification task. Congruent combinations (light background, high pitch; dark background, low pitch) enabled faster word classification among the participants. Similar results have been demonstrated in several studies investigating associations of luminance with pitch height among normal populations (Collier 2001; Hubbard 1996; Marks 1989; Melara 1989).

These findings support the idea of a relationship between pitch height and luminance as a function of general cross-modality, which has led researchers to ask whether or not these correspondences are learned. To examine this question, Walker et. al. conducted a study with infants, pairing light or dark animations with various pitches (2009). The infants looked longer at congruent combinations than incongruent combinations, which supports the hypothesis that this type of cross-modality is present very early on, and thus, potentially unlearned. Haryu and Kajikawa conducted a similar study with infants in 2012, pairing greyscale animations of apples (light grey or dark grey) with a standard tone that moved either to a lower pitch or a higher pitch. Consistent with the findings of Walker et al., the infants looked longer at congruent combinations (tone moving up with light grey apple, tone moving down with dark grey apple).

A study by Eitan, Ornoy, and Granot found that congenitally and early blind subjects associated low segments of a Beethoven sonata with “dark,” and high segments with “light,” using a five-point bipolar scale (2012). These findings suggest that blind people make cross-modal correspondences similar to those of sighted people. In contrast to the two studies involving infants, however, this may suggest that such an association is learned.

In a series of experiments varying pitch height, tempo, and pitch contour, Collier found that subjects rated high-pitched tones as not only brighter, but happier and more quickly ascending than low-pitched tones (2001). Collier also found that ascending tones were perceived in similar ways in comparison to descending tones, and discovered interactions between the three musical attributes included in his study. These associations suggest that pitch height is related not only to luminance, but to emotion, and this relationship may be shared with other visual and musical attributes.

Tonality and Luminance

Tonality (for this study’s purposes, major versus minor) is typically characterized in emotional terms borrowed from linguistics. Descriptors for minor key tonality typically include words such as “sad” and “dark,” versus major key tonality descriptors, which often include adjectives like “happy” and “light.” Suzuki et. al. (2008) used positron emission tomography in a study examining the brain areas activated by major and minor chords, and found that distinct regions were associated with the two types. Their findings suggest that the brain processes tonality in accordance with affective valence, e.g., minor chords are processed in an area of the brain which also handles emotions, but is different from the region which processes major chords.

Main effects of tonality on pleasure and surprise were observed in a 1994 study by Kellaris and Kent, in which subjects assigned higher pleasure ratings to major tonality than minor tonality. Participants also rated major music excerpts as happier than minor music excerpts in an experiment by Hunter, Schellenberg, and Schimmack (2010).

Although tonality is likely correlated with emotion, little research has been conducted to examine whether or not descriptors involving luminance, which are commonly assigned to tonality, can extend beyond linguistics and into cross-modal associations where luminance is a real visual concept, not just a metaphor. Previous studies on color and emotion have produced findings very similar to the positive/negative valence observed with regards to tonality (Boyatzis & Varghese 1994; Hemphill 1996), which supports the likelihood of a relationship between color and tonality. Furthermore, a 2004 study by Xin et. al. indicates that luminance is more relevant than some other visual attributes, such as hue, with regards to associations made between color and emotion. If a relationship between luminance and tonality exists, this may support a cognitive explanation for the linguistic metaphors so often used to characterize tonality. These metaphors include the concept of dark and light – a semantic property directly borrowed from luminance.

Suggested Reading from Inquiries Journal

Adorno’s case is built on the composers’ treatment of melodic subjects, which he views as ‘pure’ in an intellectual and philosophical sense. He contrasts this to the work of the classical and romantic composers, specifically Beethoven, whose work he views as overcommitted to the formal façade unintentionally constructed by Bach: essentially he claims that Beethoven et al rejected Bach’s challenging, philosophically... MORE»
Filmmaker Wes Anderson has developed a distinct aesthetic style that is easily recognizable through his use of striking color palettes. The purpose of this research was to draw conclusions about social stratification, social construction and the role of family as they relate to color palettes in Anderson's films. Relying on the... MORE»
This essay will discuss visual perception as well as examine the relationship between art and hallucination. The paper is divided into three sections. The first section will explore phenomenological aspects of the Ayahuasca hallucinations described in Benny Shanon‟s book Antipodes of the Mind. In the second section, it will... MORE»
Multicultural and multilingual classrooms have become the norm in many educational (and professional) settings throughout the U.S. due to changing immigration patterns caused by globalization (Institute for Educational Leadership, p. 2, 2005). Subsequently, understanding the role that culture plays in the classroom is essential... MORE»
Submit to Inquiries Journal, Get a Decision in 10-Days

Inquiries Journal provides undergraduate and graduate students around the world a platform for the wide dissemination of academic work over a range of core disciplines.

Representing the work of students from hundreds of institutions around the globe, Inquiries Journal's large database of academic articles is completely free. Learn more | Blog | Submit

Follow SP

Latest in Psychology

2017, Vol. 9 No. 03
The evolution of social media sites, such as Facebook, Twitter, Instagram, etc., have changed the way we look at relationships. Social networking sites have become a popular place to meet and connect with other people. They are also a place where... Read Article »
2017, Vol. 9 No. 02
The human memory system comprises multiple components. The focus of this paper is on the encoding and decoding process of episodic memories. Events that catch the attention of individuals either consciously or sub-consciously are stored in a... Read Article »
2017, Vol. 9 No. 02
Pica is a condition that has been prevalent among humans for centuries. According to the DSM-V (2013) pica is classified as an eating disorder in which an individual consumes non-food substances at least once per month, at a developmental stage... Read Article »
2016, Vol. 8 No. 11
This article provides a historical perspective on the bystander effect, a social phenomenon that Darley and Latané first studied experimentally in 1968. Critical events that took place prior to the study of the bystander effect are discussed... Read Article »
2016, Vol. 8 No. 11
A meta-analytic review was conducted on the adverse outcomes and prevalence of sexual harassment in a federally operated workplace. Both males and females were included in the analysis which fills a gap in prior research. Researchers report a total... Read Article »
2016, Vol. 8 No. 05
Music has accompanied major social events throughout the history of mankind. Major gatherings such as weddings, graduations, or birthdays are usually recognized by a familiar tune. There is evidence that music plays a large role in emotional processes... Read Article »
2016, Vol. 8 No. 03
Fear finds its place in the world through numerous forms, including natural disasters, genocide, war, trauma and countless other events which present lasting effects in the lives of those who maintain recollection. The research concerning the alteration... Read Article »

What are you looking for?


"Should I Go to Graduate School?"
Finding Balance in Graduate School
Presentation Tips 101 (Video)