Kyle Rankin on modern server hardening for the cloud

In this episode, I talk with Kyle Rankin, vice president of engineering operations at Final, a credit card startup. We discuss old versus new approaches to server hardening in light of the cloud, how institutional inertia thwarts change, and the new security-minded desktop OS Qubes.

Here are some highlights:

Organizational inertia and security

To me, a pretty big problem is that there are a lot of outdated approaches that just haven’t been brought up to date. I think the biggest barrier to change is inertia. If you go to a lot of orgs that have had systems around for a while, getting everyone to generate an SSH key and use it is one big thing. Another thing is, a lot of orgs have all these other security practices, like sharing group accounts, for instance; all of the developers may have one role account called ‘developer’ on all machines, and they just share the password. What ends up happening, is that you have individual pockets of people in the org who know what the right thing to do is and are frustrated that they’re not allowed to do it—a lot of times it sounds like this is a political problem. You’ll see a lot of boards that dictate password rotation, and they will demand it as a best practice, even though there’s a lot of research that shows it’s not. If you’re someone who’s in that org, and you’re not a decision maker, then you end up with things like this. There are a lot of legacy systems out there, and it’s a lot of work to update them.

The silver lining of shifting to the cloud

Many people have operated under a mistaken assumption that their internal network was secure already, and for decades, they focused on the perimeter. Then someone’s internal work station gets owned, and it’s game over. The cloud forces you to start with the assumption that the perimeter network isn’t the only network to be concerned about, that you also have the question of the internal network. You start with that assumption, and you start with the expectation that the network is somewhat hostile. Then you bake in better practices. A lot of orgs, for instance, will just use TLS externally, and won’t use it internally. In the cloud, it’s particularly important because if you don’t have your security group rules, for instance, set up correctly, you can potentially have some hostile asker pretend to be one of your servers. But if you use TLS for all of your communication, you know it’s encrypted which is nice, but more importantly, you can authenticate the server and client to each other. I know that when I’m talking to something else in the cloud, I’m talking directly to that server and not someone in between.

A new approach to desktop security: Qubes

I heard about Qubes a year or two ago. It’s essentially a way to isolate what you do on your workspace into a bunch of different VMs. I had some friends in security that had been doing that the hard way for a long time, where if they wanted to go to a bank, their bank website for instance, they would fire up a VM specifically for that purpose. I had been following a hardware startup called Purism that’s creating these open source hardware laptops that have some interesting security features. One of them is that they ship with Qubes already installed by default.

One of the big challenges with Qubes is that it heavily relies on virtualization features of your processor—you have to have well-supported hardware for it to work well. It sort of takes you back to the old days of Linux, where you had to have it installed and have people help you get it set up.

I started using it personally and quickly saw how powerful it would be for my work use. For instance, at work, what it allows me to do is tightly segment different VM’s for different purposes and colorize them. Instead of having a bunch of small windows that are all different versions of a Linux desktop, it’s just the application window you see. It feels and acts a lot like a regular Linux desktop; the difference is, I may have three different browser windows open at one time, and the borders around those windows may be red or green or blue depending on how much I trust that VM. For instance, I have an untrusted VM that has no personal files on it, and I just use it for everyday web browsing. If someone were to send me a link to look at, I would click on it and open it in my untrusted web browser. Because there are no personal files in that browser, if for some reason it were a malicious link, there’s nothing for it to compromise; it’s just the VM. If I ever suspected it were compromised, it’s relatively easy to just turn it off, erase, and create it again.

If someone sends me a PDF right now, Qubes makes it easy to integrate this into an email client. You send me a PDF attachment, and I can automatically open it in the disposable VM. If that PDF were malicious, it just compromises that temporary VM, and when I close the window, everything is erased. It allows me to calm down a little bit as far as my operational security on my desktop.