Web Audio API

Chapter 4. Pitch and the Frequency Domain

So far we have learned about some basic properties of sound: timing and volume. To do more complex things, such as sound equalization (e.g., increasing the bass and decreasing the treble), we need more complex tools. This section explains some of the tools that allow you to do these more interesting transformations, which include the ability to simulate different sorts of environments and manipulate sounds directly with JavaScript.

theory: Basics of Musical Pitch

Music consists of many sounds played simultaneously. Sounds produced by musical instruments can be very complex, as the sound bounces through various parts of the instrument and is shaped in unique ways. However, these musical tones all have one thing in common: physically, they are periodic waveforms. This periodicity is perceived by our ears as pitch. The pitch of a note is measured in frequency, or the number of times the wave pattern repeats every second, specified in hertz. The frequency is the time (in seconds) between the crests of the wave. As illustrated in Figure 4-1, if we halve the wave in the time dimension, we end up with a correspondingly doubled frequency, which sounds to our ears like the same tone, one octave higher. Conversely, if we extend the wave’s frequency by two, this brings the tone an octave down. Thus, pitch (like volume) is perceived exponentially by our ears: at every octave, the frequency doubles.

Figure 4-1. Graph of perfect A4 and A5 tones side by side

Octaves are split up into 12 semitones. Each adjacent semitone pair has an identical frequency ratio (at least in equal-tempered tunings). In other words, the ratios of the frequencies of A4 to A#4 are identical to A#4 to B.

Figure 4-1 shows how we would derive the ratio between every successive semitone, given that:

1. To transpose a note up an octave, we double the frequency of the note.

2. Each octave is split up into 12 semitones, which, in an equal tempered tuning, have identical frequency ratios.

Let’s define a to be some frequency, and to be that same note one octave higher. we know that this is the relationship between them:

Next, let k be the fixed multiplier between any two adjacent semitones. Since there are 12 semitones in an octave, we also know the following:

Solving the system of equations above, we have the following:

Solving for k:

Conveniently, all of this semitone-related offsetting isn’t really necessary to do manually, since many audio environments (the Web Audio API included) include a notion of detune, which linearizes the frequency domain. Detune is measured in cents, with each octave consisting of 1200 cents, and each semitone consisting of 100 cents. By specifying a detune of 1200, you move up an octave. Specifying a detune of −1200 moves you down an octave.

Pitch and playbackRate

The Web Audio API provides a playbackRate parameter on each AudioSourceNode. This value can be set to affect the pitch of any sound buffer. Note that the pitch as well as the duration of the sample will be affected in this case. There are sophisticated methods that try to affect pitch independent of duration, but this is quite difficult to do in a general-purpose way without introducing blips, scratches, and other undesirable artifacts to the mix.

As discussed in Basics of Musical Pitch, to compute the frequencies of successive semitones, we simply multiply the frequency by the semitone ratio 2^1/12. This is very useful if you are developing a musical instrument or using pitch for randomization in a game setting. The following code plays a tone at a given frequency offset in semitones:

function playNote(semitones) {
  // Assume a new source was created from a buffer.
  var semitoneRatio = Math.pow(2, 1/12);
  source.playbackRate.value = Math.pow(semitoneRatio, semitones);
  source.start(0);
}

As we discussed earlier, our ears perceive pitch exponentially. Treating pitch as an exponential quantity can be inconvenient, since we often deal with awkward values such as the twelfth root of two. Instead of doing that, we can use the detune parameter to specify our offset in cents. Thus you can rewrite the above function using detune in an easier way:

function playNote(semitones) {
  // Assume a new source was created from a buffer.
  source.detune.value = semitones * 100;
  source.start(0);
}

If you pitch shift by too many semitones (e.g., by calling playNote(24);), you will start to hear distortions. Because of this, digital pianos include multiple samples for each instrument. Good digital pianos avoid pitch bending at all, and include a separate sample recorded specifically for each key. Great digital pianos often include multiple samples for each key, which are played back depending on the velocity of the key press.

Multiple Sounds with Variations

A key feature of sound effects in games is that there can be many of them simultaneously. Imagine you’re in the middle of a gunfight with multiple actors shooting machine guns. Each machine gun fires many times per second, causing tens of sound effects to be played at the same time. Playing back sound from multiple, precisely-timed sources simultaneously is one place the Web Audio API really shines.

Now, if all of the machine guns in your game sounded exactly the same, that would be pretty boring. Of course the sound would vary based on distance from the target and relative position [more on this later in Spatialized Sound], but even that might not be enough. Luckily the Web Audio API provides a way to easily tweak the previous example in at least two simple ways:

1. With a subtle shift in time between bullets firing

2. By changing pitch to better simulate the randomness of the real world

Using our knowledge of timing and pitch, implementing these two effects is pretty straightforward:

function shootRound(numberOfRounds, timeBetweenRounds) {
  var time = context.currentTime;
  // Make multiple sources using the same buffer and play in quick succession.
  for (var i = 0; i < numberOfRounds; i++) {
    var source = this.makeSource(bulletBuffer);
    source.playbackRate.value = 1 + Math.random() * RANDOM_PLAYBACK;
    source.start(time + i * timeBetweenRounds + Math.random() * RANDOM_VOLUME);
  }
}

The Web Audio API automatically merges multiple sounds playing at once, essentially just adding the waveforms together. This can cause problems such as clipping, which we discuss in Clipping and Metering.

This example adds some variety to AudioBuffers loaded from sound files. In some cases, it is desirable to have fully synthesized sound effects and no buffers at all [see Procedurally Generated Sound].

Demo: to hear multiple gunshot samples played in rapid succession, visit http://webaudioapi.com/samples/rapid-sounds/.

theory: Understanding the Frequency Domain

So far in our theoretical excursions, we’ve only examined sound as a function of pressure as it varies over time. Another useful way of looking at sound is to plot amplitude and see how it varies over frequency. This results in graphs where the domain (x-axis) is in units of frequency (Hz). Graphs of sound plotted this way are said to be in the frequency domain.

The relationship between the time-domain and frequency-domain graphs is based on the idea of fourier decomposition. As we saw earlier, sound waves are often cyclical in nature. Mathematically, periodic sound waves can be seen as the sum of multiple simple sine waves of different frequency and amplitude. The more such sine waves we add together, the better an approximation of the original function we can get. We can take a signal and find its component sine waves by applying a fourier transformation, the details of which are outside the scope of this book. Many algorithms exist to get this decomposition too, the best known of which is the Fast Fourier Transform (FFT). Luckily, the Web Audio API comes with an implementation of this algorithm. We will discuss how it works later [see Frequency Analysis].

In general, we can take a sound wave, figure out the constituent sine wave breakdown, and plot the (frequency, amplitude) as points on a new graph to get a frequency domain plot. Figure 4-2 shows a pure A note at 440 Hz (called A4).

Figure 4-2. A perfectly sinusoidal 1-KHz sound wave represented in both time and frequency domains

Looking at the frequency domain can give a better sense of the qualities of the sound, including pitch content, amount of noise, and much more. Advanced algorithms like pitch detection can be built on top of the frequency domain. Sound produced by real musical instruments have overtones, so an A4 played by a piano has a frequency domain plot that looks (and sounds) very different from the same A4 pitch played by a trumpet. Regardless of the complexity of sounds, the same fourier decomposition ideas apply. Figure 4-3 shows a more complex fragment of a sound in both the time and frequency domains.

Figure 4-3. A complex sound wave shown in both time and frequency domains

These graphs behave quite differently over time. If you were to very slowly play back the sound in Figure 4-3 and observe it moving along each graph, you would notice the time domain graph (on the left) progressing left to right. The frequency domain graph (on the right) is the frequency analysis of the waveform at a moment in time, so it might change more quickly and less predictably.

Importantly, frequency-domain analysis is still useful when the sound examined is not perceived as having a specific pitch. Wind, percussive sources, and gunshots have distinct representations in the frequency domain. For example, white noise has a flat frequency domain spectrum, since each frequency is equally represented.

Oscillator-Based Direct Sound Synthesis

As we discussed early in this book, digital sound in the Web Audio API is represented as an array of floats in AudioBuffers. Most of the time, the buffer is created by loading a sound file, or on the fly from some sound stream. In some cases, we might want to synthesize our own sounds. We can do this by creating audio buffers programmatically using JavaScript, which simply evaluate a mathematical function at regular periods and assign values to an array. By taking this approach, we can manually change the amplitude and frequency of our sine wave, or even concatenate multiple sine waves together to create arbitrary sounds [recall the principles of fourier transformations from Understanding the Frequency Domain].

Though possible, doing this work in JavaScript is inefficient and complex. Instead, the Web Audio API provides primitives that let you do this with oscillators: OscillatorNode. These nodes have configurable frequency and detune [see the Basics of Musical Pitch]. They also have a type that represents the kind of wave to generate. Built-in types include the sine, triangle, sawtooth, and square waves, as shown in Figure 4-4.

Figure 4-4. Types of basic soundwave shapes that the oscillator can generate

Oscillators can easily be used in audio graphs in place of AudioBufferSourceNodes. An example of this follows:

function play(semitone) {
  // Create some sweet sweet nodes.
  var oscillator = context.createOscillator();
  oscillator.connect(context.destination);
  // Play a sine type curve at A4 frequency (440hz).
  oscillator.frequency.value = 440;
  oscillator.detune.value = semitone * 100;
  // Note: this constant will be replaced with "sine".
  oscillator.type = oscillator.SINE;
  oscillator.start(0);
}

Demo: to listen to oscillators of various types, visit http://webaudioapi.com/samples/oscillator/.

In addition to these basic wave types, you can create a custom wave table for your oscillator by using harmonic tables. This lets you efficiently create wave shapes that are much more complex than the previous ones. This topic is very important for musical synthesis applications, but is outside of the scope of this book.