How big, fast data is transforming business intelligence

In this podcast episode, I speak with Ian Fyfe, senior director for product marketing at Zoomdata, about the next generation of business intelligence software and how it addresses what Fyfe calls “the modern world of big and streaming data.”

Fyfe lists three aspects of modern data that have strained traditional BI systems:

• Big data: many companies need to quickly query billions of records, which is a scale that traditional relational databases weren’t designed to handle.
• High-velocity data streams: as more devices get connected to the internet, many applications will involve hundreds or thousands of records per second streaming into databases in real time.
• Unstructured data: relational databases were meant to handle well-structured data—integers, dates, times, short strings, and so on. Now companies need to analyze large blocks of rich unstructured data like social media postings, customer reviews, and call-center logs.

Moreover, customer demands have changed. Users of BI software have become more sophisticated, and expect to have data-exploration tools “infused into every application, every business process,” says Fyfe. And they want short feedback loops, with queries answered instantaneously—what Fyfe calls “speed-of-thought analysis” that enables real-time research. “You’re not going to use a tool if it takes more than a few seconds to run,” says Fyfe.

Fyfe also describes a couple of case studies:

• A pharmaceutical company with billions of rows of data on patients who aren’t getting potentially helpful therapies.
• An auto insurer with a vast database of historical price quotes. In the past, querying the database took so long that the company only used data to inform 3% of its new price quotes, but faster analytics have made it possible to inform every new quote with historical insight.

The business intelligence market is generally mature, Fyfe says, but four factors are pushing it into a new generation of technology:

• The increasing complexity of data sets, especially streaming data.
• The rise of embedded analytics, and the expectation that analytics and visualization will be infused into every application and business process.
• End users have become more sophisticated, and they expect to be able to conduct interactive data discovery.
• The cloud: “the business intelligence market has been a bit of a laggard in moving to the cloud,” says Fyfe, but new-generation BI software has become flexible as to whether it can run on-premise or in the cloud.

This post and podcast is a collaboration between O’Reilly and Zoomdata. See our statement of editorial independence.