A table with a variety of electroacoustic music gear. Image courtesy Blake Zidell & Associates for NYCEMF and the New York Philharmonic Biennial)
Electroacoustic Music with Video: Comparison with Sound for Film

Electroacoustic Music with Video: Comparison with Sound for Film

Electroacoustic music with video is now frequently programmed in classical concert venues. In such a concert, a video is projected on a screen and seen.  It can present changes in tempo color, shape, distance, density, and gradual or abrupt transitions.  The electroacoustic composition played with this video is projected from loudspeakers and heard. It can also present changes in tempo, pitch, loudness, distance, duration, timbre, rhythm, and gradual or abrupt transitions.  With so many parameters available in both these sense modalities for artistic coordination and relationship with each other, there are clearly many exciting and dramatic or delicate and subtle ways in which video and sound can be organized together into an effective work of art.

But after attending a number of recent concerts in which pieces for electroacoustic music with video were performed, it struck me that in many of these pieces there is only the most minimal relationship between the video and the audio.  Whatever parameter was changing or remaining static in the video was not occurring at the same rate as changes in any parameter of the electroacoustic music. The differences in their rate of change and the type of change did not seem to be patterned or deliberate.

Spoken text has the power to bridge over and meld together music and video, whatever their relationship to each other. But most of these pieces of electroacoustic music with video had no spoken text. That lack of spoken text draws attention all the more to the lack of coherent organization between the video and the audio.

But could what I experienced in those concerts of electroacoustic music with video be a deliberate aesthetic, in which the composer and video artist are consciously avoiding any tie-in with the other sensory modality?

The Theoretical Aesthetic of Independent Music and Visuals

Could there be a deliberate aesthetic in which the composer and video artist are consciously avoiding any tie-in with the other sensory modality?

Earlier examples cited of such an aesthetic often include John Cage and Merce Cunningham’s collaborations of the 1960s. These were often presumed to be in a style in which there was a deliberate non-relation of music and dance, in which neither supposedly related to the other. This aesthetic still persists in certain collaborations between composers and choreographers.

Nonetheless, in the performances of Cage and Cunningham that I attended, the two artists managed to limit and control their music and movement quite carefully, so that the random synchrony I’ve seen in current music with video seemed rarely, if ever, to occur. What I witnessed was a palpable tension between the live dancer and the sound. One could see Cunningham’s bodily resistance to falling into moving in the pattern of the music. His dancing became counter-movements that opposed the patterns being heard in the sound. This tension was for me the aesthetic they were aiming for, and was for me the most interesting part of their performances. But as far as I could tell, that interesting tension relied primarily upon Cunningham’s highly trained professional movement skills, and secondarily upon the tendency for Cage to choose a limited number of timbral possibilities in his musical accompaniment.  The limited timbres created a bland, tension-less sound palatte that helped frame the subtle tension of the dancer’s patterned muscular resistance.

While moving, the aesthetic required Cunningham to counter what was going on in the music, to not fall into the natural tendency to move his body with the music.  By moving in patterns not present in the music, Cunningham created a distinct relationship of opposites between his movements and the music. In short, the aesthetic between Cage’s music and Cunnigham’s movement was indeed a relationship, one of opposites, of avoidance of the pattern of the other modality, not a random or chance relationship.

But the aesthetic of opposing patterns is not what I observed in these recent electroacoustic concerts with video.  Instead, there was essentially no patterned relationship at all.

Weak correlation communicates irrelevance.

What we see on the video and what we hear from the live or prerecorded music is mostly unrelated to each other: not rigorously opposed, just weakly different. They are not radically different from each other and therefore there is little tension or drama evoked.  The music and video even sometimes, for brief moments, resemble each other’s degree of change, and then change into something else.  The result is weak correlation, not opposition. Opposition communicates tension and drama. Weak correlation in anything onstage or off is likely to communicate irrelevance, non-relation.

If a concert composer is using video with their music, that composer is writing a film score.

I found myself needing a more precise way to analyze what I felt is successful and what I thought might be improved upon in these works of electroacoustic music and video. The basic issue seems to be how to clarify and intensify the relationship between sound and video image.  So I looked into some concepts from the world of film sound. After all, if a concert composer is using video with their music, that composer is writing a film score.

abstract business background with media screens and silhouettes of business people holding two giant puzzle pieces

Sound and Visual Categories in Film

The three sound components used in the film industry are: speech, music, sound effects/ambient sound, and the interactions between them.  For concert electroacoustic music, we could also include sounds constructed through software synthesis or other electronic means in either the “music” or “sound effects/ ambient sound” categories, based on the dominance of identifiable pitch or unpitched noise in each sound.

Two aspects of film that may be relevant to this discussion are mise-en-scène, which is what is visible in the frame, and the way it’s arranged and shown, through editing the transitions between different shots.

Ways of Relating Video to Electroacoustic Audio, Based on Film Sound Techniques

A. The direction and depth of the image, in relation to the direction and depth of the sound

The composer has a choice when a particular image occurs on the video. He or she can:

(1.) Match the perceived direction and depth of the visual image.  When the visual image changes its position, change the direction and depth of the sound to match.

(2.) Contradict the perceived direction and depth of the visual image, by using the opposite in the sound, e.g. if the image appears far away, then have the sound be very present, close to the audience.

(3.) Maintain a constant sound placement, regardless of the direction and depth distance of the video image.  This is the current preference of the film industry, which tries to produce a continuous sound track with approximately the same sound volume and depth of distance to the microphone for every character, regardless of where the character appears.

Any of these choices could be effective in electroacoustic music with video, if applied with enough precision in the use of reverb and timbral filtering to move the sound to its intended depth.

B. Continuity of sound, as opposed to disjunction of sound: Smooth versus Rough editing

In smooth editing the sound is continuous, a flow with no sharp breaks. All of the sound edits are gradual. In rough editing or “jump cutting” the sound begins and ends abruptly, a series of sounds with sharp breaks.  All of the sound edits are abrupt or even separated by silences.

Through smooth audio editing, continuous sounds can bind together disparate images.

Through smooth video editing, continuous visuals can bind together disparate sounds.

The composer has a choice when a smooth visual flow occurs on the video. He or she can:

(1.) Match the smoothness of visual flow with the smooth editing of the sound score, resulting in a continuous flow of sound score. Any rough edges have been smoothed over.

(2.) Contradict the smoothness of visual flow with rough jump cut editing of the sound score.

The composer has a choice when a smooth visual flow alternates with jump cuts on the video.

He or she can:

(1.) Match the video’s changes from smooth visual flow to jump cuts, resulting in a smooth flow in both video and audio, which then changes into a sharp jump cuts in both video and audio.

(2.) Contradict the video’s changes from smooth visual flow to jump cuts, resulting in smooth visual flow along with rough jump cuts in the sound, and then rough jump cuts in the video along with smooth, continuous flow of sound.

(3.) Maintain a consistent smooth or a consistent abrupt editing of sound events, regardless of whether the video is moving smoothly or in jumps.

However, when sound and visual image are synchronized, that is, moving in the same tempo in the same smooth or abrupt manner, then the video movement is perceived more clearly.

C. Some other possible relationships between a sound and a video image:

Empathetic relation: The sound expresses the same emotion as the video image.

Anempathetic relation: The sound is without emotion in relation to the image.

Ironic sound: The sound expresses the opposite emotion from the image.

Surrealistic sound: Without rational connection to the video image.

Counterpoint: Loosely connected in time, a series of musical phrases of similar content is heard while a series of images of similar content, unrelated to the sounds, is seen.

Expressive editing: A sound is exaggerated and paired with a similar visual activity in an image.

Some additional references:

Zetti, Herbert. Sight, Sound, Motion: Applied Media Aesthetics, 7th Edition. New York: Cengage Learning (The Wadsworth Series in Broadcast and Production), 2013.

Chion, Michel. Audiovision: Sound on Screen. New York: Columbia University Press, 1994.

Ribrant, Gunnar. “Style parameters in film sound.” Stockholm: Stockholms Universitet Filmvetenskapliga insitutionen, 1999. (This article is available directly online as a PDF download, or through emailing the author.)



NewMusicBox provides a space for those engaged with new music to communicate their experiences and ideas in their own words. Articles and commentary posted here reflect the viewpoints of their individual authors; their appearance on NewMusicBox does not imply endorsement by New Music USA.

2 thoughts on “Electroacoustic Music with Video: Comparison with Sound for Film

  1. Marcelo Carneiro

    It was nice to read your article. I have worked on what I call video-música (and the Canadian composer Jean Piché called videomusique) on my Ph.D. Dissertation. Vídeo-Música is what you called in your article electroacoustic music with video: it’s not, as I propose, electroacoustic music, nor video, but a hybrid between both (technically and aesthetically related with electroacoustic music and video art, expanded cinema, experimental film). It has little or nothing to do, in many of its aspects, with Visual Music, as it is called a specific approach in audiovisual composition. There I propose several ways to compose with both sound images and visual images. Some of my approaches had to do with those topics you have mentioned and I am still developing, as a professor at Rio de Janeiro State Federal University, techniques for composing Vídeo-Música, and as a consequence, ways to analyze them and talk about them. If everything goes well, I will be publishing my dissertation in a book version soon. Anyone who wants to discuss and talk about this topic, I have just created a page on Facebook just for it. https://www.facebook.com/V%C3%ADdeo-M%C3%BAsica-227817493957360/

  2. Robert Scott Thompson

    This is such a great article! Wonderful, insightful and detailed. Thank you so much for taking the time to share these thoughts and perspectives.

    During the 1980’s I, together with artist Victoria Bearden, did a series of works with video and electroacoustic sound. These works often involved “multi-channel” video as a way of connecting to the multi-layered (polychronic) nature of the sound materials. We attempted to make connections and elaborations between the sonic and visual textures as a basic compositional premise. Now, some 30 years later (gasp), the works may be seen (and perhaps also heard) as somewhat quaint. We always approached our work as “film” and often composed image to the sound, rather than sound to image, as part of our practice.

    We began this work together in 1981 and, of course, these were hardscrabble analog days. In fact, we began with film, Super-8 and 16mm. Part of the current problem may be that creating video – on the desktop – and not in the camera, television studio and editing room – can lead to a kind of “disengagement” from the visual and perhaps also the aesthetics of film-making of visual art. It is now perhaps too easy to make “process” the focus of the elaboration of visual elements and, as is sometimes the case with current electroacoustic music, the composer can be lulled into easy applications of “process” without considering the materials themselves and the importance of formal design. If composers of music want to delve into visual art it might be important to consider the utility of “eye-training” – as a kind of correlate of ear-training. What is there to see? Visual narrative, form and process are “different” from their sonic counterparts, and exploring this innate difference is what makes Intermedia work so interesting. However, the stronger proposal is for the composer to collaborate with a visual artist – a video artist, graphic designer, animator or film maker.

    For those interested there is a modest DVD – Cloud Gazing



    From the liner notes:

    “These intermedia works are predicated on the notion of “kinetic electronic painting.” The video has become almost completely subverted as an artistic medium by television, and essentially languishes as a means of reportage – however obtuse the content may be. Yet, the techniques of digital image manipulation are highly congruent with those of electroacoustic music composition and sound processing – with the use of filtering, modulation, time and spectrum manipulations, and so on. It may be natural, therefore, for the electroacoustic music composer to become interested in the visual domain as enlivened by the context of video synthesis.

    My goal is to create works that might best be described as “visual music” – a synergistic melding of electronic sound and electronic image that I originally investigated beginning in 1981. Generally, my musical work tends toward abstraction in morphology and conception, and this is also true of my visual music works which eschew transparent narrativity in favor of a more enigmatic expression of shape, density, light and color in the visual domain. “

Comments are closed.