The wisdom hierarchy: From signals to artificial intelligence and beyond

We are swimming in data. Or possibly drowning in it. Organizations and individuals are generating and storing data at a phenomenal and ever-increasing rate. The volume and speed of data collection has given rise to a host of new technologies and new roles focused on dealing with this data, managing it, organizing it, storing it.

But we don’t want data. We want insight and value.

We can think about these new technologies and roles, and the way they help us move from data to insight and value, through the lens of something called the wisdom hierarchy. The wisdom hierarchy is a conceptual framework for thinking about how the raw inputs of reality (signals) are stored as data and transformed first into information, then into knowledge, and finally into wisdom. We might call these steps in the hierarchy “levels of insight.”

To illustrate, starting with the input of specific error codes coming out of a software system, we can draw out an example of each step in the hierarchy:

• Data: All the error logs from that system.
• Information: An organized report about the error codes.
• Knowledge: The ability to read a specific error report, understand it, and possibly fix the problem.
• Wisdom: An understanding of how different and seemingly unrelated errors are actually connected symptoms of a larger, underlying problem.

Whatever esoteric connotations we might associate with the word “wisdom,” we can define it here as deep subject matter expertise. From this kind of wisdom comes decision-making, business value, and leadership.

From a human perspective, the levels of this hierarchy aren’t absolute, though. The line between each level is fuzzy, and the growing insight within a particular organization doesn’t usually follow a direct path; the journey circles back on itself. For example, as knowledge about the value of data increases, the nature of data collection might change. Insights gleaned from data analysis become data themselves. Additionally, insight doesn’t simply move up the hierarchy, but comes in from all directions. There are many inputs aside from information that create knowledge, and many inputs aside from knowledge that create wisdom and business value.

From a machine learning and AI perspective, the hierarchy undergoes a weird recursion process. While humans might be said to climb the hierarchy, machines flatten it. No matter how advanced a computer system is, it is really only dealing with data. The inputs to an AI system are data; the processing of those inputs is data centric; and the outputs are, from the computer’s perspective, simply more data. But the outputs can be interpreted by humans at each level of the hierarchy. We can understand the results in terms of information, knowledge, or—if the AI is sufficiently advanced—wisdom.

From signals to data to information

The first wave of innovation in big data was simply figuring out how to store and efficiently query massive amounts of data. NoSQL and distributed database systems like Hadoop and Cassandra made this possible.

Looking back, it’s seems inevitable that these technologies grew up with web search engines. This illustrates the point that machines collapse the hierarchy so that humans can ascend it. Countless individual humans have used their own knowledge to put content (information) onto the web. Using graph databases to turn the web into data about connections, and natural language processing to turn information into data about content, made the web navigable. Humans were now able to find and use information that was previously inaccessible.

AI and data science yield information and knowledge

Much of the current excitement in AI and data science is about generating information and modeling knowledge. Classification tools—which is where machine learning really excels—generate new information. Natural language processing tools can summarize text and tell us how the writer felt about the subject. Multidimensional vector analysis can categorize inputs based on a near-limitless number of factors. Computer vision technology, combined with trained neural networks, can identify faces, street signs, or hot dogs. In other words, the data input (numbers representing pixels in an image) results in information (“this is a picture of X”).

In a way, we might say that knowledge—the knowledge of what a hot dog looks like or the knowledge of how a particular emotion manifests in text—is contained in the model or neural network that is able to produce that information. More advanced AI uses data to simulate procedural knowledge. This could be relatively simple, like finding an efficient driving route. It could also be unimaginably complex, like a self driving car. In either case, knowledge is a process enabled by data.

Moving toward wisdom

Beyond the classification or procedural knowledge available with neural networks and deep learning, lies the realm of wisdom—the deep experiential insight where one not only knows what a thing is, but also knows why, and in what context. Wisdom allows one to be reminded of things: that new thing “rings a bell,” it is like this other thing I already know… and here’s why I think so.

If you’ve ever listened to Car Talk on NPR, or just gone to a skilled auto mechanic, you’ve experienced this. An incomplete description of the problem, a poor imitation of the funny noise from under the hood, and the experts start diving into their memory, pulling up past experiences that are like the present problem.

This is where artificial intelligence is heading. We have a long way to go before we can call machines “wise,” but some AI platforms are moving in that direction. For example, Saffron, a cognitive computing platform offered by Intel, uses a “memory fabric” to enhance the functional expertise of highly skilled humans. Saffron’s reasoning, based on graph relationships, provides instances of similarity out of huge sets of unstructured data. Another example is Watson, IBM’s AI platform, which is able to answer questions posed in natural language. In both cases, and indeed with all similar technologies, these platforms extend human capabilities rather than replacing them. The machines aren’t wise, but they help make humans wiser.

Human wisdom will always be needed

Human wisdom will always be needed. Especially human wisdom that understands what the AI is doing.

I believe human beings will always be in the loop, helping us interpret streams of information and finding meaning in the numbers. We will move higher up in the food chain, not be pushed out of the picture by automation. The future of work enhanced by data will enable us to focus on higher-level tasks.

—William Ruh, chief digital officer, GE Software, from the preface to “Learning to Love Data Science,” by Mike Barlow.

Logging error reports are meaningless if the logs are never looked at, and the information generated by a search engine or a classification model can’t become knowledge if it is never used. We build data systems and AI tools to support human needs, and we will always need humans to turn the potential insights provided by technology into real value.

It is humans who build, operate, and understand the tools that turn data into information, and information into knowledge, and knowledge into wisdom. Those people bridging the gap between computer data and human wisdom will fill a variety of roles from executive leadership to low-level analyst. But what they will have in common is an understanding of the fundamental tools of data science, and the value that data-driven wisdom can create.

Machines cannot ascend the wisdom hierarchy on their own; it is beyond their capacity because they cannot intuit meaning. Without meaning, everything is just data. Humans have been ascending the wisdom hierarchy as long as we have been human, but the inputs of modern life, the masses of data we collect and accumulate, have exceeded our capacity to process on our own. Only together, humans and intelligent machines working collaboratively, can we bring the insights available from our data into the realm of wisdom that guides value creation and decision-making.

This post is part of a collaboration between O’Reilly and Intel Saffron. See our statement of editorial independence.