Bad Data Handbook

Chapter 13. Crouching Table, Hidden Network

Bobby Norton

“You were enlightened?”

“No. I didn’t feel the bliss of enlightenment. Instead… I was surrounded by an endless sorrow.”

—Yu Shu Lien describing the effects of bad data (…or something similar…) to Li Mu Bai in Crouching Tiger, Hidden Dragon

Data is good to the extent that it can be quickly analyzed to reveal valuable information. With good data, we’re able to learn something about the world, to increase revenue, to reduce cost, or to reduce risk. When data is locked up in the wrong representation, however, the value it holds can be hidden away by accidental complexity.

Let’s start our journey in the context of a seemingly simple problem faced by many IT-intensive enterprises: keeping track of who is paying for costs incurred by the business. To name but a few types of costs, consider servers, software licenses, support contracts, rent in the data center, Amazon EC2 costs, the cost of teams building software and providing support; the list goes on. As a business grows and the costs associated to run the business increase, sooner or later, someone will ask, “What are we paying for?”

From a business owner or executive perspective, not every department within a business uses shared assets to the same degree. As such, it doesn’t make sense to evenly split costs across the entire business. Doing so may hide the fact that a particular line ...