Susan Sons on maintaining and securing the internet’s infrastructure

In this episode, O’Reilly’s Mac Slocum talks with Susan Sons, senior systems analyst for the Center for Applied Cybersecurity Research (CACR) at Indiana University. They discuss how she initially got involved with fixing the open source Network Time Protocol (NTP) project, recruiting and training new people to help maintain open source projects like NTP, and how security needn’t be an impediment to organizations moving quickly.

Here are some highlights:

“Help. I need a sysadmin.”

It all started in February of 2015 when the NTP implementation maintainer, Harlan Stenn, came to me. Among NTP’s many problems, there was a build box, and the entire build server—the entire build system—depended on this one server in Harlan’s home continuing to function. Harlan no longer had the root password for the system, couldn’t update it, didn’t know what scripts were running on it, and no one in the world could build NTP without this server continuing to function. As I was helping him, I was seeing the state of the code and infrastructure, and I found out exactly how deep the rabbit hole went. It was a moment of panic. ‘If I don’t fix this, the internet is going to fall down, finance is going to fall down, and a lot of Krypton Security is going to stop working and be very attackable. We’re already having major DDoS problems because no one’s fixed this.’ I figured out a long time ago that if there’s an emergency you’re seeing and no one else is fixing it, that means you’re in charge.

Recruiting to save the internet

The terrifying thing about infrastructure software in particular is that paying your internet service provider (ISP) bill covers all the cabling that runs to your home or business, the people that work at the ISP and their routing equipment, power, billing systems and marketing, but it doesn’t cover the software that makes the internet work. That is maintained almost entirely by aging volunteers, and we’re not seeing a new cadre of people stepping up and taking over their projects. What we’re seeing is ones and twos of volunteers who are hanging on but burning out while trying to do this in addition to a full-time job, or are doing it instead of a full-time job and should be retired, or are retired. It’s just not meeting the current needs.

Early- and mid-career programmers and sysadmins say, ‘I’m going to go work on this really cool user application. It feels safer.’ They don’t work on the core of the internet. Ensuring the future of the internet and infrastructure software is partly a matter of funding (in my O’Reilly Security talk on saving time, I talk about a few places you can donate to help with that, including ICEI and CACR) and partly a matter of recruiting people who are already out there in the programming world to get interested in systems programming and learn to work on this. I’m willing to teach. I have an Internet Relay Chat (IRC) channel set up on freenode called #newguard. Anyone can show up and get mentorship, but we desperately need more people.

Building for speed and security

Security only slows you down when you have an insecure product, not enough developer resources, not enough testing infrastructure, not enough infrastructure to roll out patches quickly and safely. When your programming teams have the infrastructure and scaffolding around software they need to roll out patches easily and quickly—when security has been built into your software architecture instead of plastered on afterward, and the architecture itself is compartmented and fault tolerant and has minimization taken into account—security doesn’t hinder you. But before you build you have to take a breath and say, ‘How am I going to build this in?’ or ‘I’m going to stop doing what I’m doing, and refactor what I should have built in from the beginning.’ That takes a long view rather than short-term planning.

Working from first principles

The single biggest issue we’re facing right now in the security industry is that we are pushing things that make good sound bites over things that are good first-principle security. Whenever you have a situation where not enough people understand the issues and there’s a lot at stake and a lot of money moving around, there is a tendency to try to sound cool and be easy to absorb and make people feel safe instead of getting good work done. I hate to break it to you, but really good engineering is rarely sexy. Fixing the pipes is rarely sexy. Often, the best things to do don’t make good sound bites. If we teach more people to work from first principles and have more mature discussions, then we can actually get our C-suite or leadership involved because we can talk in concepts that they understand instead of just talking about what firewall rules we need.