Designing with Sound

Chapter 1. Opportunities in Sound Design

THERE ARE MANY OPPORTUNITIES to improve the auditory experience in products. With additive sound, we can include sounds where there were none before, helping to change mood, increase energy, or instill a sense of calm. With subtractive sound, we can remove sound in whole or in part, making for quieter cafés, better-sounding conversations, and more pleasant products.

Sound plays an increasingly exciting role in product design, human–machine interaction, our buildings and spaces, and the quality of our notifications. We can use audio to deliver information in a way that visuals cannot, and there are many places to apply these skills. Expect to see innovations across different markets in the near future.

This chapter covers past, present, and upcoming advances in sound design and discusses often-overlooked areas where sound can enhance our relationship with the world. We begin by exploring how sound can affect our sense of taste.

Sound Design and Taste Modulation

It is a well-known fact that food tastes different in airplanes. Cabin noise actually alters your sense of taste. The low humming sounds reduce sweet and salty experiences, and amplify bitter and umami flavors (umami is a Japanese word that describes the fifth basic taste found in savory flavors, such as rich broths, meats, and shiitake mushrooms). These factors make food less tasty during air travel. This scientific theory, called taste modulation, reveals that high frequencies can heighten sweet flavors, while low frequencies can bring out bitter flavors.

To combat this, Finland’s Ultra Nordic Agency worked with China’s Master Chef Steven Liu and molecular gastronomy professor, Anu Hopia, to craft mealtime soundscapes for passengers on Finnair flights. The idea was to use elements of sound design, like the fact that high frequencies enhance our perception of sweetness, to restore some of food’s tastiness in the air. The chef and gastronomist collaborated with sound designers to record Nordic landscapes and high-frequency ranges to match the food served on the flights. Passengers were given headphones and sounds to play during their meals.

After significant testing, researchers found that these passengers enjoyed a heightened dining experience, exhibiting a greater sense of taste with the soundtrack than without. The work was nominated for multiple awards at the 2018 International Sound Awards in Hamburg.

Restaurants and researchers are experimenting with the effects of sound on the palate, too. One London restaurant, House of Wolf (now closed), served a chocolate-covered bittersweet toffee with a phone number. Dialers were given an option of pressing 1 for sweet or 2 for bitter, and a soundtrack played that enhanced each taste accordingly. Other restaurants have considered how soundtracks might enhance the specific flavors of food. This principle could be applied to wine tastings, cafeteria food, and even health improvement efforts, such as using high frequencies as a substitute for high sugar content.¹

Wireless Power Through Sound Waves

Using ultrasound, beamforming technology, optical lasers, and very specially designed transducers, Meredith Perry and her startup, uBeam, is innovating the remote transmission of energy through the air—using high-frequency sound outside of the range of human and animal hearing to beam power to wireless devices.² “Sound like sci-fi? uBeam is on the bleeding edge of technology, developing a system that wirelessly transports energy and data over distance through repetitive and precise vibration of the air—what we know as sound. Through our journey building this technology, I’ve fallen in love with the power of sound and its applicability across seemingly endless dimensions,” says the founder.³ The beam can work at up to 10 feet in line of the ultrasound beam, but they are working to expand the range further. Ultrasound has been used safely for nearly a century, and is extremely safe at very low power levels. There is no cumulative effect either, so it can work well for domestic applications. Accoding to Perry, “This is about a paradigm shift. If you’re moving from your car to a coffee shop to work and your phone is charging while you’re using it, it’s no longer about what percentage you’re at. You could stay at 1% all day.”⁴ The idea is that one day, hidden ultrasonic beaming stations could invisibly power any device in the IOT era (or “Internet of Things”), even electric vehicles. Running out of battery would be a thing of the past.

Additive Sound Design

Modern cars have engines that are so efficient, they no longer need to sound like they did in the ’40s, ’50s, and ’60s. Most car sounds are designed in some way, from the crumpf of the door closing to the sound of acceleration. In fact, automobiles are one of the most carefully sound-designed products in the world, with many absorbing, diffracting, and vibration-isolating materials added to them to ensure a quiet ride at high speeds. After spending so many decades removing sounds and making cars quieter, however, vehicle manufacturers have had to design some sounds back in. Muscle cars and hot rods sound strange when they’re quiet, so rich artificial engine sounds are designed, digitized, and piped in through vehicle speaker systems. Because engine noises are also an important safety mechanism for pedestrians, electric vehicles have artificially added engine noise and backing-up sounds to ensure that passersby hear them. Otherwise, the quietness of these efficient cars presents a potential safety hazard.

Subtractive Sound Design

The sounds from leaf blowers, large trucks, and microwaves interrupt human lives. Companies like Dyson and Blendtec invest millions in research for everyday products that produce less noise and reap the benefits in markets. Blendtec released a blender that was so quiet it could operate at volumes lower than a conversation, revolutionizing the sound of an everyday task. Dyson worked with materials scientists, engineers, and sound designers to create a quieter, hair dryer, pushing the industry forward for the first time in decades. Sometimes the most important improvement that can be made on a product is to remove some of the beeps and tones it makes in order to create a more pleasant experience. Subtractive sound design can be applied to hundreds of everyday products and environments.

Sonification

In 1908, Hans Geiger and his grad student Walther Müller filled a tube with an inert gas low-pressure and applied high voltage. This device detects radiation by producing an auditory click when radioactive particles ionize the gas and make the tube momentarily conductive. This elegant solution allowed engineers to detect invisible radiation by making it audible, transforming an entire scientific practice by allowing scientists to measure alpha, beta and gamma radiation with a relatively cheap and robust device. The development of the Geiger counter remains one of the most successful and important steps in the history of sonification: the process of converting information into sound.

Sound is strong in ways that visuals are not. Sound is more quickly perceived than sight, touch, smell, or taste.⁵ Results of many studies show that auditory reaction time is far faster than visual reaction time.⁶ Your ears can unambiguously pick out differences that your visual sense might confuse. New York Times writer Amanda Cox used this idea to sonify the last few seconds of four different 2010 Olympic events.⁷ Visually, it seemed unclear who crossed the finish line first, because the participants finished only a few milliseconds apart. By assigning a clear sound to the moment when each person crossed the finish line, however, Cox was able to create an audible story about who won.

Because sound is such an accurate way to discern differences, there are many opportunities to apply sonification to better understand our world. These kinds of systems can help turn dynamic datasets, such as stock market changes or news stories, into a less intrusive auditory sense. When done well, they can help inform professions that have an overwhelming amount of visual data to process, such as medical personnel or researchers. Sonification is beneficial for any application that overwhelms the visual sense, such as large datasets that exhibit patterns over periods of time longer than the average human attention span. Imagine being able to listen to an experiment to determine if it is going well instead of peering persistently through a microscope. Audio-based experiences could enable doctors and nurses, investors, and industrial workers to work without distraction. Sonification could also provide farmers, technicians, and factory workers with an intuitive feel or where things stand and where problems might be arising.

Sonification and Income Data on the New York Subway Line

Brian Foo, a New York–based programmer and visual artist, used 2011 US Census data to sonify income differences along the route of the New York 2 train’s 49 stations (see Figure 1-1). Foo used a total of 63 sounds for the composition, many of them from artists and musicians from areas along the subway line. He also included the NYC subway chime, a sound familiar to anyone riding the subway.

Figure 1-1. Sonification for the 2 train. The composition is minimal as the train starts toward downtown Manhattan. (Source: https://vimeo.com/118358642.)

The resulting composition matched the feeling of traveling along the subway, starting with a sensation of sleepiness as the train moves away from the city, followed by a feeling of excitement, and finally a sense of calming back down as the train leaves the city (see Figures 1-2 and 1-3).

Figure 1-2. The sound file for the 2 train reveals the overall compositional structure of the piece. (Source: https://vimeo.com/118358642.)

Figure 1-3. The soundscape of the 2 train entering Manhattan. The composition built up to more energy and complexity with the increase in income along the subway route. (Source: https://vimeo.com/118358642.)

Unlike many “fast” sonification methods, Foo used more instruments and a faster tempo for increases in income level, not pitch. He made sure to match the general composition to the data, with a flow that starts off relaxed and becomes more intense and “full” as the subway passes through the affluent financial district, and then thins out as it leaves Manhattan and enters the Bronx. It is possible to “feel” the income at each stop, and understand how it increases. The soundscape matches the visuals, the locations, and the dataset.

Foo strived to avoid assigning certain sounds to, or passing judgment on, income levels or neighborhoods. He wanted to keep the data presentation agnostic. Instead of positive or negative tones, he assigned more instruments and a faster tempo for higher income, and reduced the number of instruments and tempo with lower income.

Foo also used a method called phase shifting, a compositional approach pioneered in the 1960s by Terry Riley and Steve Reich in which two or more identical melodies are repeated with slightly variable tempos, so the melodies slowly shift in and out of sync with each other. Foo felt that this method provided the perfect metaphor for the NYC subway: “constantly looping but at different tempos, always running but never on time, phasing between order and chaos.”⁸ All 63 instruments and sounds used in the composition employ phase shifting.

sonification and Agriculture

We’re able to glean a lot about our environment from simply listening to it, and we can learn from those soundscapes to design our own. Sonification could work alongside greenhouse sensors to indicate data necessary to ensure the health of plants. For example, the greenhouse could play a minimal soundscape to provide information on the overall balance of pH, nitrogen, or soil level. The data could then be converted into signal processing that would inform the caretakers so they can adjust the missing element rebalancing the system.

Sonification and Stock Trading

Sound is underutilized in the world of financial trading and there’s an opportunity here for sound to inform and utilize peripheral attention. Day traders often work with multiple screens and a stressful visual channel. Even a pared-down, minimal display requires significant attention. Sonifying important information in the market can help reduce this visual burden.

Sound can help with emotional monitoring, too. Intense emotional swings can cause traders to panic and make poor decisions. Sound can notify traders when their heart and breathing rate, blood pressure, or galvanic skin response might signal an unreliable emotional state, potentially saving both money and emotional pain.

Sonification could also alert traders when patterns of price changes that occur in one group of correlated equities might affect the probability of high-volume trading in another. Because sound can be monitored passively, trading sonification could relieve investors from having to constantly check back and forth between monitors to review the trends, and mitigate the cognitive cost of reading visual displays.

Sound Design and Calm Technology

Calm Technology was a framework developed by Xerox PARC researchers Mark Weiser and John Seely Brown in the early 1990s as a way of reducing the cognitive impact of information on the human brain. Technology can provide information to help us make decisions, and the way in which it does so can be overwhelming or polite (i.e., calm). Weiser wrote that we need smarter humans, not smarter devices. He predicted that attention would be the scarcest resource in the future, and how technology draws on attention would make or break our interactions with it. The way to achieve a sense of calm is by informing without drawing attention away from the current task. Making technology calm is a matter of empowering the periphery.

If good design is about getting you to your goal with the fewest moves, calm technology is about getting you to your goal with the least amount of attention (see Figure 1-4).

Figure 1-4. The areas of attention around a person. Primary denotes high-resolution attention—good for desktop computing, mobile technology, and other tasks that demand full attention. Secondary denotes medium resolution attention—good for rearview mirrors and other tasks that can be done by switching from primary to peripheral attention. Tertiary denotes low resolution—best for small indicator lights and sounds. Sound and movement occupies an ambient awareness at the lowest level of attention.

Soundscapes

Brian Eno coined the term ambient music to describes a kind of music meant to influence the mood of the environment without demanding the listener’s full attention. When properly designed, sounds can profoundly shape the behaviors of large groups of people, allowing a central actor to quickly communicate something to a large group of people and coordinate them. Music is sometimes pressed into this kind of service. At shopping malls throughout China, “Going Home” by saxophonist Kenny G is played to indicate closing hours. In Taiwan, garbage trucks play a recording of Beethoven’s Für Elise as a cue for neighborhood residents to bring their garbage out for pickup. Numerous other examples exist on a small scale, but each leverages the same advantages of recorded music: it is specific, recognizable, and unignorable, but also relatively unobtrusive.

The downside, of course, is the risk of a formerly beloved song becoming irritating through constant repetition, creating a negative association with both the song and the event. Anyone who grew up with ice cream trucks playing “It’s a Small World” every day of summer is familiar with this phenomenon.

A better solution in these situations is to use generative audio (see the next section) to vary the sound. This provides all the benefits of recognizability and unobtrusiveness, but adds enough variety to avoid irritation.

Generative Audio

Generative audio is a wonderful way to introduce variation and create a constantly evolving sound. This improves human attention. Also called “procedural audio,” it is “non-linear…created in real time according to a set of programmatic rules and live input.”⁹

You can create generative audio in many different ways:

Linguistic/structural generative audio

Uses generative grammars, like the autocomplete for text messages and Gmail, to form a decision tree that can determine what might come next in a musical piece. Like language, while there are rules for what should come after something else, they are not strict, creating variability. You can create audio with an infinite amount of variability based on the rules given.

Interactive/behavioral generative audio

Doesn’t start with inputs, but uses human behavior, such as movement or other interaction with the system, to create music. The system may be previously set up, and interaction from people can modify the music in different ways. For instance, some malls have installation pieces where people can generate music and visuals by walking by at various speeds.

Creative/procedural generative audio

Describes music that is generated by processes that are designed or initiated by the composer. This may mean prerecorded pieces that are combined in particular ways.

Biological/emergent generative audio

Creates music that is infinite and cannot be repeated. An example in nature is the nonrepeating notes created by wind blowing through wind chimes.

Localized Sound

Audio beamforming is an advanced way of targeting sound to a specific location, even to the point where someone standing in the targeted area can hear the sound and someone standing two feet away can hear no sound at all. Beamforming works by using an array of speakers and real-time digital signal processing to direct sound to specific areas, while cancelling sound everywhere else. This technique would find a natural application to play audio tracks in an otherwise quiet museum, where patrons can stand in a marked spot and listen to a description of a particular piece of art, or stand outside of it to enjoy the silence. By replacing the need for a handheld audio device, this novel approach would improve hygiene and simplify the experience.

Beamforming can also be used the same way regional temperature control functions in some vehicles, allowing different listeners to control the volume of their experience separately from the other listeners in the room, for instance, for a television. Beamforming works by using an array of speakers and real-time digital signal processing to direct sound to specific areas.

Beamforming could be a big help in hospitals, where patients are often forced to listen to the medical alerts of others and where doctors and nurses experience frequent alert fatigue. What if a nurse could instantly understand the status of a patient just by walking into a room and listening to—instead of looking at—the status of all of the machines? Alerts could be delivered to contextual, noninterruptive locations in such environments. See Chapter 4 for a detailed discussion of alert fatigue in hospitals.

While beamforming is not a new concept, in the past, it has been expensive in terms of both speakers and computing resources. Improvements in the cost of computing has created affordable signal processing chips which will help bring it to a larger range of applications.

Sound as Therapy

Music therapy has been used in hospitals, schools, and rehabilitation programs, as it has been shown to help some patients struggling with anxiety, depression, and addiction. Sound as therapy has powerful applications, from helping with autism to treating cancer, post-traumatic stress disorder, and Alzheimer’s. It can also be safer to use than potentially addictive painkillers, or can be used alongside such medications to reduce their dosage. Sound not only has applications for stress relief and recovery, but it can also help rewire the brain after trauma, overcome speech difficulties, and improve memory.

One example, called NurMut, applies sound to help dementia patients suffering from disorientation, anxiety, and stress. Wearable skin sensors are calibrated to detect the onset of an anxiety attack, using an algorithm that responds to heart rate, movement, and skin conductance. In response to an oncoming attack, the system plays music to calm or cheer the patient.

Sound healing techniques may involve specially tuned singing bowls, singing or chanting, and guided meditation. The singing bowls are often made from machined metal or hand-carved crystal. Their size, shape, and composition allow them to create deep, resonant sound waves in low and high frequencies.

House cats purr at a frequency of 26 Hz, the low end of the 25–250 Hz range produced by felines overall, usually in the key of D, which is known to be soothing. These frequencies promote bone growth and tissue regeneration, leading scientists to hypothesize that cats evolved the purr to stimulate healing while conserving energy and resting. Similar benefits in humans are being investigated: “If scientists were able to mimic the purring of cats, and mechanically stimulate bones and muscles, even when a person was at rest, then the sound [and] vibrations could help induce bone growth, mitigate osteoporosis, and help with muscle loss [and] atrophy in elderly patients or even astronauts in space who face bone density and muscle loss due to weightlessness.”¹⁰ One study by NASA using a plate vibrating at 90 Hz found that 10 minutes of therapy per day helped rats maintain near-normal bone density when they were prevented from bearing weight on their hind legs (in comparison, rats who received no treatment lost 92% of normal bone density, and rats that were allowed to bear weight for 10 minutes per day lost 61% of bone density).¹¹ Studies are currently underway to test similar acoustic therapies on patients with osteoporosis. One day, purring hospital beds could help patients heal.

Conclusion

Sound is an ever-expanding frontier with great depth and surprising applications. There are enormous opportunities to add, reduce, or modify sounds in our environments. By considering sound as a crucial stimulus that can affect our degree of satisfaction, our emotions, and our focus, we can work on designing better sounds that help us instead of work against us.

¹ Anne-Sylvie Crisinel and Charles Spence, “As Bitter as a Trombone: Synesthetic Correspondences in Nonsynesthetes Between Tastes/Flavors and Musical Notes,” Attention, Perception, & Psychophysics 74, no. 7 (1994): 1994–2002, https://www.ncbi.nlm.nih.gov/pubmed/20952795.

² Adam Grant, In Originals, “Different is Normal”

³ “The Power of Sound,” https://ubeam.com/Blog/the-power-of-sound/

⁴ Marco della Cava, “uBeam’s Meredith Perry shows her stealth wireless charging technology really works,” USA Today, http://bit.ly/2Jm3WY3.

⁵ “The Speed of Hearing,” MED-EL, https://blog.medel.com/the-speed-of-hearing/.

⁶ See e.g. Jose Shelton and Gideon Praveen Kumar, “Comparison Between Auditory and Visual Simple Reaction Times,” Neuroscience and Medicine 1, no. 1 (2010): 30–2, https://www.scirp.org/journal/PaperInformation.aspx?PaperID=2689.

⁷ Amanda Cox, “Fractions of a Second: An Olympic Musical,” The New York Times, https://nyti.ms/2yAwKY5.

⁸ Brian Foo, “Two Trains,” Data-Driven DJ, https://datadrivendj.com/tracks/subway/.

⁹ Andy Farnell, “An Introduction to Procedural Audio and Its Application in Computer Games,” http://cs.au.dk/~dsound/DigitalAudio.dir/Papers/proceduralAudio.pdf.

¹⁰ “Cat Purring as a Means for Healing Bones,” The Infinite Spider, https://infinitespider.com/cat-purring-and-healing-bones/.

¹¹ “Good Vibrations,” NASA Science, https://science.nasa.gov/science-news/science-at-nasa/2001/ast02nov_1/.