Pedagogical Approaches in Music and Audio Education for Deaf and Hard-of-Hearing Students

Abstract

Learning about music, sound or audio can present significant challenges for individuals who are deaf and hard of hearing (DHH). Given the advancements in technology and the increasing emphasis on equality, diversity and inclusion (EDI) in education, this article proposes pedagogical approaches aimed at facilitating the learning process for DHH students in the areas of music and audio production. These approaches encompass sound visualisation, haptic feedback, automated transcription, tactics in non-linear editing and digital signal processing. Importantly, these approaches do not necessitate advanced technical skills or substantial additional resources, thus lowering barriers for DHH students to overcome challenges in music and audio production. Furthermore, these strategies would enable content creation and editing for individuals with DHH, who may have previously been excluded from participating in music and audio production. Recommendations are provided for the implementation of these approaches in diverse educational settings to promote the integration of EDI in music and audio education.

1. INTRODUCTION

Recent advancements in information and communication technology have led to the widespread integration of technology in workplace environments, along with a growing emphasis on fostering equality, diversity and inclusion (EDI). Collaborative efforts involving creative technologists, communication specialists and practitioners from various disciplines have sought to merge these two aspects through the development of assistive technologies and adaptive systems. By drawing upon insights from diverse fields such as neuroscience, psychology and artificial intelligence to facilitate inclusive design (Lim, Giacomin and Nickpour 2021; Nketia, Amso and Brito 2021; Skowronek, Gilberti, Petro, Sancomb, Maddern and Jankovic 2022), these innovations help promote EDI values and enhance individuals’ quality of life by empowering them to become more active contributors to society. Among these endeavours, there have been technological advancements that address the needs of audio perception and music enjoyment for individuals with varying degrees of hearing loss. The most common ones are hearing aids and cochlear implants, enabling deaf and hard-of-hearing (DHH) children to understand speech, attend mainstream schools and utilise spoken language as their primary means of communication (Zanin and Rance 2016). There are also endeavours to prototype non-audio assistive technologies, such as speech-to-text applications (Shezi and Ade-Ibijola 2020) and haptic-assistive technologies (Fletcher 2021), for assisting DHH individuals in sound creation and music performance (Trivedi, Alqasemi and Dubey 2019). These devices facilitate the translation of musical notes into alternative sensory inputs and vice versa, with the ultimate goal of making music accessible and meaningful to everyone.

The history of music education for DHH individuals predates the advent of any modern technology. As early as 1802, during the time when Beethoven completed his second symphony, a French otologist and deaf educator utilised various musical instruments intended to develop auditory discrimination of students with DHH (Hummel 1971; Solomon 1980). Notable scholarly contributions to the field of music education for DHH were made by William Wolcott Turner and David Ely Bartlett (Turner and Bartlett 1848), pioneers who advocated for music education and documented methods for teaching music to individuals with DHH (Darrow and Heller 1985). A significant milestone in the field of education for individuals with DHH came with the introduction of the Total Communication (TC) approach in the 1970s by Holcomb (1970). This educational philosophy emphasises the use of all communication modalities based on the specific needs of an individual child at a particular stage of development (Moores 2012), paving the way for more adaptive pedagogical approach and ensuring DHH students can better engage with and benefit from music learning (Knapp 1980).

Existing literature has proposed various pedagogical approaches facilitating DHH students to engage in music appreciation (Chao-Fernandez, Román-García and Chao-Fernandez 2017), develop instrumental skills (Hash 2003), participate in ensemble activities and learn music as an academic subject in school (Fawkes and Ratnanather 2009). These innovative pedagogical approaches and music practices continue to evolve, offering learning opportunities for individuals with varying levels of hearing loss and nurturing talented musicians who excel beyond their hearing (in)abilities (Cleall 1983; Glennie, Gilman and Kim 2019).

While efforts have been made to promote the integration of EDI in music education, these initiatives have primarily focused on conventional musicianship training within the classical music tradition. To our knowledge, there is limited documentation of attempts to develop competencies in contemporary music and audio production, which encompass areas such as sound design and theory, digital signal processing, audio recording, editing, mixing, and practical skills related to the use of digital audio workstations (DAWs). These knowledge and skills are closely linked to the use of computer technology and are essential for the career development of musicians in this digital age. To advocate for the integration of EDI in electronic music practices and shed light on the implementation of critical pedagogies in music education, this article presents pedagogical approaches developed by the authors to facilitate the learning process of students with DHH in music and audio production. Based on the authors’ teaching experiences and scholarly efforts, with supporting literature that aligns with the proposed pedagogy, these approaches encompass sound visualisation, haptic feedback, automated transcription, tactics in non-linear editing, and digital signal processing. Importantly, these approaches do not require technical expertise and can be implemented without the need for additional resources. Suggestions for the adoption of these approaches in the context of electroacoustic music and other educational settings will also be provided.

2. CONCEPTUAL FRAMEWORK

The pedagogical approaches presented in this article are underpinned by the social model of disability. According to this model, DHH is viewed as an interaction ‘between impairment and the surrounding social world, rather than being an individual medical problem’ (Emens 2012: 214). Restrictions faced by individuals with DHH in their social interactions are a result of institutional forms of exclusion and cultural attitudes deeply rooted in the society, which fails to provide equitable social and structural support meeting their specific needs (Terzi 2004). In contrast to the dominant medical model, which considers disability as a problem to be remedied to conform with normative standards (Paley 2002), the social model perceives disability as a societal issue rather than solely an individual problem (Cameron 2009). The social model is also different from the cultural model of disability, with the latter advocating for a ‘similar and different’ view that recognises and celebrates disability as part of our lives (Devlieger, Rusch and Pfeiffer 2003). Apart from these models of disability, there are scholarly efforts and perspectives from DHH communities advocating that deafness should not be considered a form of disability (Lane 2002; Harvey 2013). A spectrum of alternative terms, including ‘marginal’, ‘bicultural’ and ‘handicapped’, are used to describe the identity of individuals with DHH (Stumer, Hickson and Worrall 1996; Chapman and Dammeyer 2017).

Historically informed performance practice in music necessitates physical capabilities and sensorimotor skills to manipulate musical instruments, which can pose challenges for individuals with certain forms of functional diversity (Howe 2016). Moreover, established instrumental designs and rigid school music curricula have limited flexibility to accommodate specific needs (Challis 2018; Rizzo 2022), further exacerbating the inequality in music and audio education for DHH learners. As a result, these learners may lack the ability to conform to socially constructed aesthetic values in certain musical cultures and contexts (Lubet and Hofmann 2006).

Meeting the needs of DHH students in the music classroom or teaching studio may require substantial changes in both the social and the physical environment (Siebers 2008). In addition to modifying the physical space, it is essential to address the social dynamics between students and teachers to counteract the perpetuation of socially constructed injustices. Lubet (2009) argues that music teachers should actively engage in political actions against systemic ableism in various educational contexts. Bell (2017) suggests exploring alternative approaches in the design of music teaching activities. Rather than adhering strictly to historically informed instrumental practices, music education can be reimagined to incorporate digital musical instruments (DMIs) and hacking activities (Bell 2015; Bell, Bonin, Pethrick, Antwi-Nsiah and Matterson 2020), which offer increased accessibility for all learners. This sentiment is echoed by Landy (2012), who advocates for music education to embrace and celebrate the diversity of musical expressions.

3. RELATED WORKS

Music instruction for individuals with different levels and types of hearing loss continues to face practical challenges, largely due to misconceptions surrounding the ability of individuals with DHH to hear and appreciate music (Darrow 1985). Earlier studies have highlighted that instrumental instructors often exhibited hesitancy in actively recruiting DHH students, primarily stemming from a lack of familiarity with the musical capabilities of deaf learners (Darrow and Gfeller 1991), or concerns that these musicians may have a negative impact on ensemble performance quality (Sheldon 1997). Music educators have often perceived individuals with hearing loss, alongside other behaviourally or emotionally disadvantaged students, as the most challenging and exceptional populations within mainstream classroom music education (Gfeller, Darrow and Hedden 1990).

Darrow (1993) discovered that individuals with hearing loss derive enjoyment from singing or signing songs, listening to music, and engaging in movement and dance with music, similar to their hearing counterparts. She suggested that individuals with hearing loss often exhibit a stronger sense of rhythm than pitch-related abilities, and are more proficient in discriminating lower frequency ranges compared with higher ranges (Darrow 2007). While DHH students perceive and communicate music differently from others, they can still find pleasure in music (Chen-Hafteck and Schraer-Joiner 2011; Vaisberg, Martindale, Folkeard and Benedict 2019). Research has shown that individuals with DHH are more capable of recognising rhythm and tone duration than pitch and melody, while their ability to differentiate between timbres is generally diminished (McDermott 2004; Drennan and Rubinstein 2008; Looi, McDermott, McKay and Hickson 2008). Vaisberg, Beaulac, Glista, Macpherson and Scollie (2021) investigated the preferred frequency-gain shaping for speech and music perception in individuals with cochlear implants. Their study found that low-frequency gain was significantly increased relative to the prescription for speech and music stimuli, and that gain adjustments varied among listeners. The findings also revealed that music preferences were driven by changes in perceived fullness and sharpness, while speech preferences were driven by changes in perceived intelligibility and loudness. The study suggested that prescribed amplification to optimise speech intelligibility and alternative amplification for music perception should be recommended for most listeners. This implies that a prescribed and personalised audio workstation could be a viable approach to facilitating the learning process of DHH students in music and audio production.

Current practices often incorporate other sensory inputs, such as visual and tactile cues, to address the challenges faced by individuals with DHH (Trivedi et al. 2019; Deja, Torre, Lee, Ciriaco and Eroles 2020; Hopkins, Maté-Cid, Fulford, Seiffert and Ginsborg 2021). These approaches have shown effectiveness in musicianship training across different levels of hearing loss. However, the competencies required for music producers and sound designers extend beyond the conventional musicianship framework and encompass skills such as design thinking, computer literacy and studio practices (Alsop and Berry 2009). This indicates that teaching and learning approaches in music production and sound design can differ significantly from those in general music education (Hug and Kemper 2014). While some existing practices in music education can be adopted, the strategies proposed in this article focus on sound processing and audio editing within DAWs and other related hardware and software to enhance the learning and workflow of music production and sound design.

4. SOUND VISUALISATION

Sound visualisation has proven to be a valuable assistive tool for audio recording and editing, particularly in optimising volume and timbre (Lima, Dos Santos, and Meiguins 2021). It also serves as a useful tool to enhance situational awareness for people with DHH (Azar, Saleh, and Al-Alaoui 2015), compensating for the absent of or reduced auditory perception. To facilitate the workflow of DHH audio engineers, guidance can be prepared by other teammates providing information about the desired levels, relative distances between sources and the microphone, reflective surfaces and background noise sources prior to recording. This allows a DHH audio engineer to choose appropriate recording devices and estimate input levels and gain without relying on real-time auditory monitoring. Background noise levels can be visually displayed on a meter to ensure they remain below the desired sound sources. If audio metering is not available in the hardware device being used, there are numerous assistive mobile apps that can serve as substitutes. Many of these apps are free and can be connected to the audio source via line or auxiliary (Aux) audio output, with display format configurations typically available without much effort. A test recording is always recommended, so that recorded output can be visually assessed to identify any issues such as a high noise floor and excessive reverberation. Impulsive sounds are recommended for testing purposes, particularly indoors, to identify and minimize any undesired eigentones using equalisation or downward expansion techniques during post-production processing and sound-effect editing. Visual cues can also assist in identifying suitable edit points and confirm the source, intensity and nature of interactions depicted in filmed content. When using an external microphone, the best setup can be achieved by maintaining a hand’s distance between the microphone and the sound source. This ensures off-axis delivery and helps prevent plosives, which can be tested by pronouncing the letter ‘P’ in front of the microphone.

In different stages of production, various types of sound visualisation can provide valuable information. The peak programme meter (PPM) is commonly used to measure volume levels and identify instances of clipping. Waveforms, displayed in the sound editing software, illustrate the dynamic range of audio, aiding in determining the appropriate amount of compression to apply. Spectrograms, although less commonly used in everyday audio engineering, can be helpful in identifying unwanted sounds and enhancing specific resonances.

Visual matching of content can also be employed to ensure consistency with other takes or reference audio files. Some software offers visual analysis of audio clips, providing measurements such as root mean square (RMS) and peak and loudness unit full scale (LUFS). This information is useful for ensuring consistent volume levels among different sources, particularly when working with voices of different genders and age groups, as the spectral content may vary significantly, making it challenging to hear certain voices in comparison.

5. HAPTIC FEEDBACK

Haptic feedback refers to the use of mechanical vibrations to provide alternative sensory feedback. An early example of this approach can be found in the rumoured story of Beethoven, who modified his piano by removing its legs to allow for the perception of sound through tactile vibrations felt at floor level (Wallace 2018). Technological advancements nowadays have made haptic feedback more approachable, and numerous affordable haptic devices have been made available that can translate sound into vibrations. These devices can be particularly useful in sound editing and mixing, as they enhance the perception of timing and facilitate the differentiation and similarity of signals in a temporal manner akin to sound itself (Beattie, Frier, Georgiou, Long and Ablart 2020). The field of haptic technology is continuously evolving, and the use of ultrasound for haptics has opened up new possibilities, including the potential for depth perception (Morales, Marzo, Freeman, Frier and Georgiou 2021).

In addition to haptic devices, haptic feedback can also be achieved through the vibrations of loudspeaker cones. Touching the loudspeakers is not uncommon in the audio production process, which allows DHH audio engineers to perceive the relative timing and intensity of different signals, particularly for the extended bass content. Furthermore, with the growing trend of science, technology, engineering, art and mathematics (STEAM) and hacking, individuals with DHH have the opportunity to develop their own do-it-yourself (DIY) haptic feedback devices (Andreotti and Frans 2019; Bell et al. 2020). This DIY approach empowers individuals to explore and customise haptic solutions according to their specific needs and preferences.

6. NON-LINEAR EDITING

Non-linear editing involves the manipulation and arrangement of audio clips or tracks in a non-sequential or non-linear manner, providing flexibility in editing and manipulating audio recordings beyond on-site and real-time production processes. This approach allows for the integration of assistive tools to facilitate the workflow of audio engineers and sound designers. Such tools include context-aware editing platforms for text-based speech (Morrison, Rencker, Jin, Bryan, Caceres and Pardo 2021), algorithms for sound source recognition (Yang 2021), and neural networking-based plug-ins for automatic compression (Singh, Bromham, Sheng and Fazekas 2021). However, these assistive tools may not fully capture the underlying emotions conveyed in the text or the applied sound effects in the soundtracks. To address this issue, editors can utilise mirrors to confirm the mouth shapes of voice actors and interpret the emotional intent of the dialogue through lip reading. In cases where non-linear editing involves voiceover recordings with separate sound and motion, editors can keep track of word start and stop times and sentence completion to facilitate editing and ensure natural speech pacing. Additionally, visual references can be employed to verify the timing of audio effects. By capturing video simultaneously with sound-effect recording, the audiovisual material can be imported into the video editing software, allowing for concurrent viewing of visual content alongside sound. The visual reference provides valuable information for the sound designer regarding the timing and source of sound effects, which can be hidden in the final export.

While many of the aforementioned approaches are applicable to individuals with different types of DHH, there are specific strategies that can enhance the workflow for those who are hard of hearing but not completely deaf. These individuals may have significant hearing loss but can still be able to understand speech through auditory processes and/or special adaptations (Heward 2006). In non-linear and non-destructive editing, for instance, temporary pitch shifting can be employed to lower the audio content down into an audible range for a sound editor with upper frequency hearing loss. Once the editing process is complete, the audio content can then be restored to its original frequency ranges. Destructive editing procedures can be logged in the pitch-shifted copy and then applied to the original audio content to avoid degradation.

7. AUTOMATED TRANSCRIPTION

After the non-linear editing process, the clarity of the dialogue can be reaffirmed through automated transcription. Any discrepancies between the script and the transcribed text can then be identified for further editing. However, cultural and environmental factors can impact the accuracy of transcription. For example, the English dialogue may sound correct locally in a non-English-speaking country but may not be accurately transcribed in a particular transcription platform. Additionally, the choices for non-English languages may be limited in those transcription services. This is coupled with other issues such as the differences in speech and writing, and the impact of background sounds on the accuracy of transcription.

Many online video sharing and social media platforms, as well as certain video editing software, offer the feature of automatic captioning for user-uploaded videos. Numerous studies in the existing literature have explored the application and effectiveness of automatic captioning in areas such as language learning and deaf culture (e.g. Smith, Crocker and Allman 2017; Perez 2022). DHH sound editors can utilise auto-captioning services to identify poorly pronounced dialogue and take follow-up actions. This process is particularly valuable in editing dialogue in animation and games, where retakes of the voice-overs are common.

In addition to speech-to-text captioning, automated transcription is also available for certain musical materials. Established techniques in music information retrieval (MIR) enable accurate transcription of recorded music into organised and structured musical data, such as MIDI and XML files (Benetos, Dixon, Duan and Ewert 2019). These technologies have the capability to separate music clips into distinct components such as instrumental tracks (Kumar, Biswas and Roy 2020), extract melodies sourced from different timbres (Hernandez-Olivan, Zay Pinilla, Hernandez-Lopez and Beltran 2021), and detect the genre of a particular piece of music (Estolas, Malimban, Nicasio, Rivera, Pablo and Takahashi 2020). Many of these applications are commercially available as standalone software or DAW plug-ins (Benetos et al. 2019). Parameters of these musical elements can then be identified through visual or haptic feedback, enabling DHH sound engineers to edit music based on their technical knowledge. In some cases, it may even be possible to make changes to the musical content of a recorded performance. Operating systems, such as those from Apple and Microsoft, are gradually incorporating sound analysis and peripheral functions into their integrated development environments (IDEs), which will lead to the availability of more accessible technologies beyond their current applications primarily in the industrial and healthcare sectors.

8. DIGITAL SIGNAL PROCESSING

Noise reduction can pose challenges as the sensitivity to noise varies among individuals with different levels of hearing loss (Heinonen-Guzejev et al. 2011). DHH audio engineers and sound designers can employ various techniques to address this issue. They can record sample audio clips of background ambience beforehand and compare them to the recording of the actual take to identify the noise floor and spectral balance of typical sound sources through sound visualisation tools. Compression and downward expansion can be applied to optimise levels and reduce noise with visual reference to waveform displays. Similarly, sound equalisation can be optimised by comparing spectrograms. De-essing can be achieved using a similar approach to the aforementioned noise reduction techniques. Pronunciation of sibilant letters such as ‘F’, ‘X’ and ‘S’, as well as other soft consonants, can lead to a significant increase in levels that require correction. These peaks can be identified through waveform displays and by identifying candidate words using subtitles.

While mastering is often outsourced to specialists, smaller studios and projects may involve audio engineers handling it themselves. In this case, DHH audio engineers may face challenges, as their hearing conditions can affect their precision in critical listening. However, advancements in audio technology have made digital mastering more accessible and affordable compared with analogue mastering. The inclusion of digital signal processing has led to the development of sophisticated algorithms and software tools that automate various mastering processes (Birtchnell 2018), including parametric equalisation and compression/expansion. These parameters can be optimised by reviewing and benchmarking the final mix with reference material such as broadcasted programmes and commercially available music products, using spectrograms, waveforms displays and metering.

9. IMPLICATIONS FOR EDUCATORS

In the previous section, we proposed various approaches to facilitate the learning process of DHH students in audio production and sound design, including sound visualisation, haptic feedback, automated transcription, tactics in non-linear editing and digital signal processing. These approaches are not overly demanding in terms of technical skills or resources and can be effectively integrated into different educational contexts and curricula. While there have been efforts to promote EDI in the fields of electroacoustic music and sonic art, it is important to also consider the specific needs of DHH students within the constraints of the mainstream music curriculum in school learning environments. The increasing integration of technology in school music education presents opportunities for the design and implementation of more accessible and inclusive pedagogical approaches (Ruthmann and Manite 2018), as proposed in this article.

Additionally, it is beneficial to select music that incorporates varied rhythms and tone durations, as these elements are particularly accessible to DHH students (McDermott 2004; Drennan and Rubinstein 2008; Looi et al. 2008), which can help enhance inclusiveness in the school music teaching and learning environment. If feasible, music with more pronounced drum and bass elements can be options for students with upper frequency hearing loss. The sequence of skills taught in the lesson content can be optimised to better accommodate DHH students. For example, techniques such as pitch shifting and band filtering can be taught prior to other DAW skills, which facilitates a smoother learning progression for those students and nurtures their aesthetic awareness and creativity (Daniel 2020).

In higher education contexts, such as undergraduate and postgraduate degree programmes in sound design and music production, alternative teaching and learning arrangements can be arranged for students with different levels of hearing loss. These include the development of self-directed learning resources that incorporate the aforementioned approaches, tailored practical tasks and assessment methods for DHH students. Detailed rubrics can be created in consultation with students and programme leaders, and advice can be sought from the university’s equality and diversity officers, if necessary. A successful example can be found in the Accessible Podcasting programme ratified by the Canadian Hard of Hearing Association and offered by Seneca Polytechnic,¹ where the second author serves as an advisory committee member.

In addition to the pedagogical approaches outlined in this article, the successful integration of DHH students into music and audio education can be further facilitated through teacher training and professional development in special needs education (Wolf and Younie 2019). It is essential to provide instructional support in the classroom environment and ensure that music objectives are clearly defined, with appropriate resources available to support DHH students (Darrow and Gfeller 1991). The development of an inclusive audio education and music curriculum is crucial to enhancing the effectiveness and appropriateness of learning content for students with hearing loss and other special learning needs, thus promoting a broader integration of EDI (Gouge 1990). These aspects are vital in creating an inclusive and supportive learning environment that caters to the diverse needs of all students, including those with hearing loss.

10. CONCLUDING REMARKS

This article contributes to the current body of knowledge by challenging established norms in mainstream music and audio education, presenting pedagogical approaches for inclusive teaching and learning practices. These approaches aim to make music and audio education more accessible and inclusive for DHH students without requiring significant additional resources. By modifying teaching practices, DHH students can learn to become sound engineers, producing audio and musical content for their hearing counterparts, who form the majority of consumers in the music industry and media sector. They can also create content that is meaningful to their community, a contribution that may not be achieved by hearing sound engineers. Moreover, some of these approaches can benefit other students by providing alternative perspectives and ensuring the quality of their work. This fosters mutual understanding between students with and without hearing loss, thereby promoting a more inclusive sonic environment.

In the broader context of electroacoustic music studies, the proposed pedagogical approaches have the potential to empower DHH individuals to advance their sonic practices and promote the wider integration of EDI. This integration should encompass the inclusion and representation of marginalised groups across all dimensions of identity, including individuals with DHH who may have often been left out of the equation when it comes to music and sound (Gouge 1990). By embracing EDI, the fields of electroacoustic music and sonic art can flourish as diverse and vibrant spaces that value and celebrate the contributions of all individuals, ensuring that functional diversity is not overlooked in the realm of music and sound.