DevOps for finance

Common Challenges

From small trading firms to big banks and exchanges, financial industry players are looking at the success of Google and Amazon for ideas on how to improve speed of delivery in IT, how to innovate faster, how to reduce operations costs, and how to solve online scaling problems.

Financial services, cloud services providers, and other Internet tech companies share many common technology and business challenges.

They all deal with problems of scale. They run farms of thousands or tens of thousands of servers, and thousands of applications. No bank—even the biggest too-big-to-fail bank—can compete with the number of users that an online company like Facebook or Twitter supports. On the other hand, the volume and value of transactions that a major stock exchange or clearinghouse handles in a trading day dwarfs that of online sites like Amazon or Etsy. While Netflix deals with massive amounts of streaming video traffic, financial trading firms must be able to keep up with low-latency online market data that can peak at several millions of messages per second, where nanosecond accuracy is necessary.

These Big Data worlds are coming closer together, as more financial firms like Morgan Stanley, Credit Suisse, and Bank of America adopt data analytics platforms like Hadoop. Google (in partnership with SunGard) is one of the shortlisted providers bidding on the Securities and Exchange Commission’s new Consolidated Audit Trail (CAT), a secure platform that will hold the complete record of every order, quote, and trade in the US equities and equities options markets: more than 50 billion records per day from 2,000 trading firms and exchanges, all of which needs to be kept online for several years. This will add up to several petabytes of data.

The financial services industry, like the online tech world, is viciously competitive, and there is a premium on innovation and time to market. Businesses (and IT) are under constantly increasing pressure to deliver faster, and with greater efficiency—but not at the expense of reliability of service or security. Financial services can look to DevOps for ways to introduce new products and services faster, but at the same time they need to work within constraints to meet strict uptime and performance service-level agreements (SLAs) and compliance and governance requirements.

DevOps Tools in the Finance Industry

DevOps is about changing culture and improving collaboration between development and operations. But it is also about automating as many of the common jobs in delivering software and maintaining operating systems as possible: testing, compliance and security checks, software packaging and configuration management, and deployment. This strong basis in automation and tooling explains why so many vendors are so excited about DevOps.

A common DevOps toolchain includes:

• Version control and artifact repositories
• Continuous Integration/Continuous Delivery servers like Jenkins, Bamboo, TeamCity, and Go
• Automated testing tools (including static analysis checkers and automated test frameworks)
• Automated release/deployment tools
• Infrastructure as Code: software-defined configuration management tools like Ansible, Chef, CFEngine, and Puppet
• Virtualization and containerization technologies such as Docker and Vagrant

Build management tools like Maven and Continuous Integration servers like Jenkins are already well established across the industry through Agile development programs. Using static analysis tools to test for security vulnerabilities and common coding bugs and implementing automated system testing are common practices in developing financial systems. But as we’ll see, popular test frameworks like JUnit and Selenium aren’t a lot of help in solving some of the hard test automation problems for financial systems: integration testing, security testing, and performance testing.

Log management and analysis tools such as Splunk are being used effectively at financial services organizations like BNP Paribas, Credit Suisse, ING, and the Financial Industry Regulatory Authority (FINRA) for operational and security event monitoring, fraud analysis and surveillance, transaction monitoring, and compliance reporting.

Automated configuration management and provisioning systems and automated release management tools are becoming more widely adopted. CFEngine, the earliest of these tools, is used by 5 of the 10 largest banks on Wall Street, including JP Morgan Chase. Puppet is being used extensively at the International Securities Exchange, NYSE, E*Trade, and the Bank of America. Bloomberg, the Standard Bank of South Africa (the largest bank in Africa), and others are using Chef. Electric Cloud’s automated build and deployment solutions are being used by global investment banks and other financial services firms like E*Trade.

While most front office trading systems still run on bare metal in order to meet low latency requirements, Docker and other containerization and virtualization technologies are being used to create private clouds for testing, data analytics, and back office functions in large financial institutions like ING, Société Générale, and Goldman Sachs.

Financial players are truly becoming part of the broader DevOps community by also giving back and participating in open source projects. For example, LMAX, who we will look at in more detail later, has open sourced its automated tooling and even some of its core infrastructure technology (such as the low-latency Disruptor inter-thread messaging library). And at this year’s OSCON, Capital One released Hygieia, an open source Continuous Delivery dashboard.

Financial Operations Is Not WebOps

Financial services firms are hiring DevOps engineers to automate releases and to build Continuous Delivery pipelines, and Site Reliability Engineers (patterned after Google) to work in their operations teams. But the jobs in these firms are different in many ways, because a global bank or a stock exchange doesn’t operate the same way as Google or Facebook or one of the large online shopping sites. Here are some of the differences:

• Banks or investment advisers can’t run continuous, online behavioral experiments on their users, like Facebook has done. Something like this could violate securities laws.
• DevOps practices like “Monitoring as Testing” and giving developers root access to production in “NoOps” environments so that they can run the systems themselves work for online social media startups, but won’t fly in highly regulated environments with strict requirements for testing and assurance, formal release approval, and segregation of duties.
• Web and mobile have become important channels in financial services—for example, in online banking and retail trading—and web services are used for some B2B system-to-system transactions. But most of what happens in financial systems is system-to-system through industry-standard electronic messaging protocols like FIX, FAST, and SWIFT, and low-latency proprietary APIs with names like ITCH and OUCH. This means that tools and ideas designed for solving web and mobile development and operations problems can’t always be relied on.
• Continuous Deployment, where developers push changes out to production immediately and automatically, works well in stateless web applications, but it creates all kinds of challenges and problems for interconnected B2B systems that exchange thousands of messages per second at low latencies, and where regulators expect change schedules to be published up to two quarters in advance. This is why this book focuses on Continuous Delivery: building up automated pipelines so that every change is tested and ready to be deployed, but leaving actual deployment of changes to production to be coordinated and controlled by operations and compliance teams, not developers.
• While almost all Internet businesses run 24/7, most financial businesses, especially financial markets, run on a short trading day cycle. This means that a massive amount of activity is compressed into a small amount of time. It also means that there is a built-in window for after-hours maintenance and upgrading.
• While online companies like Etsy must meet PCI DSS regulations for credit card data and SOX-404 auditing requirements, this only affects the “cash register” part of the business. A financial services organization is effectively one big cash register, where almost everything needs to be audited and almost every activity is under regulatory oversight.

Financial industry players were some of the earliest and biggest adopters of information technology. This long history of investing in technology also leaves them heavily weighed down by legacy systems built up over decades; systems that were not designed for rapid, iterative change. The legacy problem is made even worse by the duplication and overlap of systems inherited through mergers and acquisitions: a global investment bank can have dozens of systems performing similar functions and dozens of copies of master file data that need to be kept in sync. These systems have become more and more interconnected across the industry, which makes changes much more difficult and riskier, as problems can cascade from one system—and one organization—to another.

In addition to the forces of inertia, there are significant challenges and costs to adopting DevOps in the financial industry. But the benefits are too great to ignore, as are the risks of not delivering value to customers quickly enough and losing them to competitors—especially to disruptive online startups powered by DevOps. We’ll start by looking at the challenges in more detail, to understand better how financial organizations need to change in order for them to succeed with DevOps, and how DevOps needs to be changed to meet their requirements.

Then we’ll look at how DevOps practices can be—and have been—successfully adopted to develop and operate financial systems, borrowing ideas from DevOps leaders like Etsy, Amazon, Netflix, and others.