97 Things Every Data Engineer Should Know

Chapter 39. How to Prevent a Data Mutiny

Sean Knapp

Founder and CEO of Ascend

Data teams are on a collision course. We’ve seen growth in the volume, velocity, and variety of data that leads to more-complex systems, but the most important component of a company’s data sophistication is by far the number of people—architects, engineers, analysts, and scientists—able to build new data products, and their efficiency in doing so.

As enterprises scale their efforts and their teams to build new data products, the interconnectedness and resulting complexity can be paralyzing for these groups. Worse still, they often have conflicting priorities, sending them down a path wrought with conflict. For infrastructure teams, building for scale, security, and cost are of the utmost importance, while engineering teams prioritize flexibility, development speed, and maintainability. Meanwhile, data scientists and analysts are focused on availability and discoverability of data, and connectivity of tools.

Catering to just one group’s needs is a guaranteed strategy to incite a “data mutiny,” in which internal users create shadow IT organizations with a mandate to move quickly and free themselves from the conflicting priorities. However, new processes and technologies can help bring back this balance of speed and flexibility, without risking scale, security, and maintainability.

With DevOps, we enabled ...