97 Things Every Data Engineer Should Know

Chapter 1. A (Book) Case for Eventual Consistency

Denise Koessler Gosnell, PhD

Consider the life of owning a bookstore.¹ At some point, you will want to set up a system that maintains an inventory of all the books in your store.

In the store’s early days, you selected a system that was designed for about 1,000 books at one location. This system updates a book’s record during a customer’s checkout. When a customer approaches the counter to buy a book, your inventory system does the following:

1. Checks your ledgers for inventory details

2. Records the new purchase

3. Updates all the records

4. Returns a receipt to the customer

The receipt confirms both the customer’s purchase and that your inventory system is up-to-date. This type of inventory system requires all of your transactions to have strong consistency. In this sense, strong consistency refers to all accesses to the store’s inventory being processed sequentially and read from the same state in your store’s inventory system.

Well, good news, store owner! The book business is booming, and you are opening multiple stores to address your growing customer base. In this world, how do you maintain your company’s inventory across multiple stores?

Maybe you consider rearchitecting your current system. You decide to have one master register in your first store. Then, all of your other stores will have a new register that connects to the master.

This new system works great...until you lose power to your master register. Now all of the customers at all of your stores are blocked from making a single purchase. Your lines are long and stagnant.

Are you going to let your customers walk away frustrated when your queue takes too long to process? To resolve this, maybe you decide that validating and updating your store’s inventory can be done in a different way.

Maybe this time you consider updating your ledgers on a daily basis. In this new system, at nightly close, you update the inventory of your books by counting all of them in each store and validating that each title you expect is on your shelves. Each night, your stores’ shelves pass their tests, and you can go home resting assured that your books are balanced.

This second type of inventory system plans for all of your transactions to have eventual consistency. Each access to your store’s inventory is processed in parallel and logs an update to your book’s inventory system. Eventually, all accesses about a particular book will return the last updated value.

And what happens if a customer goes to look for a title that isn’t on your shelves? Well, you can address it then.

Addressing inconsistencies when they are found is like read repair in a distributed system. Only when you find an inconsistency do you kick off the heavier process to update state across all of your ledgers. During this process, you can inspect your sales logs to see whether the records are up-to-date. The result of these multiple inspections of state is to ensure that your records are up-to-date and consistent.

Is this process good enough for you? For systems that have to trade high-volume processing for state consistency, yes.

Swapping to an eventually consistent system with read repair to address inconsistencies will keep your customers from the peril of long lines at the register. And it will keep your architects feeling less stress about protecting the availability of your master register’s database.

Loosely speaking, the two approaches to keeping inventory in your store represent two schools of thought. They describe two different architectures: one that is strongly consistent (addressing scale with client/server–style architectures) and another that is eventually consistent (handling high-volume transactions and resolving stateful inconsistencies when they are found).

So for the system you are designing, what school of thought do you want to plan with?

¹ Denise Gosnell would like to thank David Bechberger for his contributions to this chapter.