Chapter 9. Models for Sequence Analysis

Analyzing Variable-Length Inputs

Up until now, we’ve worked only with data with fixed sizes: images from MNIST, CIFAR-10, and ImageNet. These models are incredibly powerful, but there are many situations in which fixed-length models are insufficient. The vast majority of interactions in our daily lives require a deep understanding of sequences—whether it’s reading the morning newspaper, making a bowl of cereal, listening to the radio, watching a presentation, or deciding to execute a trade on the stock market. To adapt to variable-length inputs, we’ll have to be a little bit more clever about how we approach designing deep learning models.

Figure 9-1 illustrates how our feed-forward neural networks break when analyzing sequences. If the sequence is the same size as the input layer, the model can perform as expected. It’s even possible to deal with smaller inputs by padding zeros to the end of the input until it’s the appropriate length. However, the moment the input exceeds the size of the input layer, naively using the feed-forward network no longer works.

Feed-forward networks thrive on fixed input size problems. Zero padding can address the handling of smaller inputs, but when naively utilized, these models break when inputs exceed the fixed input size.

Figure 9-1. Broken feed-forward network

Not all hope ...

Get Fundamentals of Deep Learning, 2nd Edition now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.