Automation Is a Must

Any solution that involves fiddling with every one of your machines by hand is almost always the wrong one. This book will make that approach a thing of the past for you.

Even in the best of economic climates, system administrators always have too much to do. This is true both for the people who do this work by choice and for those who had a boss walk into their office and say, “Hey, you know about computers. We can’t hire anyone else. Why don’t you be in charge of the servers?” When hiring gets frozen, existing employees (including those not trained for the task) may well be asked to take on added system administration responsibilities.

Automation, when applied intelligently, is one of the few things that can actually make a difference under these circumstances. It can help people work more efficiently, often freeing up time previously spent on sysadmin scut work for more interesting things. This can improve both productivity and morale.

My editors and I changed the title of this edition of the book because we realized that the real value of the material was its ability to make your life better through automation. In this book, I’ll try very hard to give you the tools you need (including the mental ones—new ways to think about your problems, for example) to improve your time at work (and, as you’ll see in the last chapter, your time at play).

How Perl Can Help You

System administrators should use any and every computer language available when appropriate. So why single out Perl for a book?

The answer to this question harks back to the very nature of system administration. Rémy Evard, a colleague and friend, once described the job of a system administrator as follows:

On one side, you have a set of resources: computers, networks, software, etc. On the other side, you have a set of users with needs and projects—people who want to get work done. Our job is to bring these two sets together in the most optimal way possible, translating between the world of vague human needs and the technical world when necessary.

System administration is often a glue job, and Perl is one of the best glue languages. Perl was being used for system administration work well before the World Wide Web came along with its voracious need for glue mechanisms. Conversations I’ve had with numerous system administrators at Large Installation System Administration (LISA) conferences and other venues have indicated that Perl is still the dominant language in use for the field.

Perl has several other things going for it from a system administration perspective:

It has visible origins in the various Unix shells and the C language, which are tools many system administrators are comfortable using.
It is available on almost all modern operating systems and does its best to present a consistent interface on each. This is important for multiplatform system administration.
It has excellent tools for text manipulation, database access, and network programming, which are three of the mainstays of the profession.
The core language can easily be extended through a carefully constructed module mechanism.
A large and dedicated community of users has poured countless hours into creating modules for virtually every task. Most of these modules are collected in an organized fashion (more on these collections in a moment). This community support can be very empowering.
It is just plain fun to program.

In the interest of full disclosure, it is important to note that Perl is not the answer to all of the world’s problems. Sometimes it is not even the appropriate tool for system administration programming. There are a few things going against it:

Perl has a somewhat dicey object-oriented programming mechanism grafted on top of it. Python or Ruby is much better in this regard.
Perl is not always simple or internally self-consistent and is chock-full of arcane invocations. Other languages have far fewer surprises.
Perl is powerful and esoteric enough to shoot you in the foot.

The moral here is to choose the appropriate tool. More often than not, Perl has been that tool for me, and hence it’s the focus of this book.

This Book Will Show You How

In the 1966–68 Batman television show, the dynamic duo wore utility belts. If Batman and Robin had to scale a building, Batman would say, “Quick Robin, the Bat Grappling Hook!” or “Quick Robin, the Bat Knockout Gas!” and they’d both have the right tool at hand to subdue the bad guys. This book aims to equip you with the utility belt you need to do good system administration work.

Every chapter attempts to provide you with three things:

Clear and concise information about a system administration domain

In each chapter, we discuss in depth one domain of the system administration world. The number of possible domains in multiplatform system administration is huge; there are far too many to be included in a single book. The best survey books on just Unix system administration— Essential System Administration by Æleen Frisch (O’Reilly), and Unix System Administration Handbook, by Evi Nemeth, Garth Snyder, Scott Seebass, and Trent H. Hein (Prentice Hall)—are two and three times, respectively, the size of this book, and we’ll be looking at topics related to three different operating systems: Unix (including variants like Linux), Windows-based operating systems, and Mac OS X.

The list of topics covered is necessarily incomplete, but I’ve tried to put together a good stew of system and network administration information for people with varying levels of experience in the field. Seasoned veterans and new recruits may come away from this book having learned completely different material, but everyone should find something of interest to chew on. Each chapter ends with a list of references that can help you get deeper into a topic should you so choose.

For each domain or topic—especially those that have a considerable learning curve—I’ve included appendixes that will give you all the information you need to get up to speed quickly. Even if you’re familiar with a topic, you may find that these appendixes can round out your knowledge (e.g., showing how something is implemented on a different operating system).

Perl techniques and approaches that can be used in system administration

To get the most out of this book, you’ll need some initial background in Perl. Every chapter is full of Perl code that ranges in complexity from beginner to advanced levels. Whenever we encounter an intermediate-to-advanced technique, data structure, or idiom, I’ll take the time to carefully step through it, piece by piece. In the process, you should be able to pick up some interesting Perl techniques to add to your programming repertoire. My hope is that Perl programmers of all levels will be able to learn something from the examples presented in this book. And as your Perl skills improve over time, you should be able to come back to the book again and again, learning new things each time.

To further enhance the learning experience, I will often present more than one way to accomplish the same task using Perl, rather than showing a single, limited answer. Remember the Perl motto, “There’s more than one way to do it.” These multiple-approach examples are designed to better equip your Perl utility belt: the more tools you have at hand, the better the choices you can make when approaching a new task.

Sometimes it will be obvious that one technique is superior to the others. But this book addresses only a certain subset of situations you may find yourself in, and a solution that is woefully crude for one problem may be just the ticket for another. So bear with me. For each example, I’ll try to show you both the advantages and the drawbacks of each approach (and often tell you which method I prefer).

System administration best practices and deep principles

As I mentioned at the start of this chapter, there are better and worse ways to do system administration. I’ve been a system and network administrator for the past 25 years in some pretty demanding multiplatform environments. In each chapter I try to bring this experience to bear as I offer you some of the best practices I’ve learned and the deeper principles behind them. Occasionally I’ll use a personal “war story from the front lines” as the starting point for these discussions. Hopefully the depth of the craft in system administration will become apparent as you read along.

What You Need

To get the most out of this book, you will need some technical background and some resources at hand. Let’s start with the background first:

You’ll need to know some Perl: There isn’t enough room in this book to teach you the basics of the Perl language, so you’ll need to seek that information elsewhere before working through this material. A book like Learning Perl , by Randal L. Schwartz et al. (O’Reilly), can get you in good shape to approach the code in this book.
You’ll need to know the basics of your operating system(s): This book assumes that you have some facility with the operating system or systems you plan to administer. You’ll need to know how to get around in that OS (run commands, find documentation, etc.). Background information on the more complex frameworks built into the OS (e.g., WMI on Windows or SNMP) is provided.
You may need to know the specifics of your operating system(s): I’ll attempt to describe the differences between the major operating systems as we encounter them, but I can’t cover all of the intra-OS differences. In particular, every variant of Unix is a little different. As a result, you may need to track down OS-specific information and roll with the punches should that information be different from what is described here.

For technical resources, you will need just two things:

Perl

You will need a copy of Perl installed on or available to every system you wish to administer. The downloads section of the Perl website will help you find either the source code or the binary distribution for your particular operating system. The code in this book was developed and tested under Perl 5.8.8 and ActivePerl (5.8.8) 822. See the next section for more information about these versions.

The ability to find and install Perl modules

A later section of this chapter is devoted to the location and installation of Perl modules, an extremely important skill for our purposes. This book assumes you have the knowledge and necessary permissions to install any modules you need.

At the end of each chapter is a list of the version numbers for all of the modules used by the code in that chapter. The version information is provided because modules are updated all the time, and not all updates retain backward compatibility. If you run into problems, this information can help you determine whether there has been a module change since this book was published.

Some Notes About the Perl Versions Used for This Book

I chose to develop and test the code in this book under Perl 5.8.8 and ActivePerl (5.8.8) 822. These choices might lead you to ask a few questions.

What About Perl 5.10?

The Perl 5 development team has done some fabulous work to produce 5.10. They’ve added some great features to the language that I encourage you to explore. However, 5.10 wasn’t released until well after this edition was under way, and at the time of this writing no major OS distribution has shipped with it as its default version of Perl. Because I know the adoption of new versions takes a while, I didn’t want to include code in the book that depended on features in the language most people couldn’t use out of the box. All of the code here should work just fine on Perl 5.10, and in the interest of making this code useful to as many readers as possible, I deliberately chose to target the previous stable release.

What About Strawberry Perl?

Strawberry Perl is an effort to bring a more “generic” and self-sufficient version of Perl to the Win32 platform. ActiveState’s Perl distribution ships with a packaging system (PPM) so users don’t have to compile modules or update them via the Comprehensive Perl Archive Network (CPAN). Strawberry Perl aims to provide an environment where compilation and CPAN use are easy (or at least possible) and are the norm.

I think this is an excellent project because it is helping to push some portability back into the non-Win32 Perl community. Some great progress has been made so far, but the project is still fairly young as of this writing and it does not yet have a sufficiently large ecosystem of available modules (e.g., lots of the Win32:: modules are missing). That ruled it out for this edition, but it is definitely something to watch.

What About Perl 6?

Ah, that’s the big question, isn’t it? I have the pleasure of occasionally bumping into Jesse Vincent, the current Perl 6 project manager (and author of the fabulous RT trouble ticketing system). Here’s what he had to say when I asked about Perl 6:

Perl 5 is a mature, widely deployed, production-ready language. Perl 6 is maturing rapidly, but isn’t yet ready for production deployment.

There are some Perl 5 modules that let you get a taste of some planned Perl 6 features (some of which have found their way into Perl 5.10). I encourage you to try modules like Perl6::Slurp and Perl6::Form. But at this point in time, there just isn’t a language implementation ready for production use, and hence there is no Perl 6 in this book. Furthermore, once Perl 6 is ready for widespread use, it will take considerable time for the necessary ecosystem of modules to be developed to replace the many, many modules we leverage in this book. I look forward to that time; perhaps you’ll see a Perl 6 edition of this book some day.

Some Notes About Using Vista with the Code in This Book

The code in this book has been tested under Microsoft Vista, but there is one twist you will need to know about if you plan to use it on that platform: some of the examples in this book must be run using elevated privileges for this to work. Which things require this and which don’t is somewhat idiosyncratic. For example, part of the Windows quota example in Chapter 2 works without elevated privileges and part (the important part) fails with an unhelpful error if it doesn’t have them.

Under Vista’s User Account Control (UAC), it is not enough to be running the code as an Administrator; you must have explicitly requested it to run at an elevated privilege level. Here are the ways I know to run Perl scripts at that privilege level (since you can’t by default right-click and use “Run as administrator”). You should choose the method or methods that make the most sense in your environment:

Use the runas.exe command-line utility.
Designate that the perl.exe binary itself be run as an Administrator (right-click on the binary name, choose Properties, switch to the Compatibility tab, and select “Run this program as administrator.”
Use one of the Elevation Power Toys described at http://technet.microsoft.com/en-us/magazine/2008.06.elevation.aspx and http://technet.microsoft.com/en-us/magazine/2007.06.utilityspotlight.aspx to allow Perl scripts to be Run as administrator.
Use the command-line utility pl2bat to convert your Perl script into a batch file and then permit that batch file to run as Administrator. Batch files don’t require any special magic (like the previous option) for this to happen.

You may be wondering if it is possible to add something to your Perl script to have it request elevated privileges as needed. Unfortunately, according to Jan Dubois (one of the top Windows Perl luminaries in the field), the answer is no. He notes that there is no way to elevate an already running process; it must be created with elevated privileges. The closest you could come would be to check whether the process was already running in this fashion (e.g., by using the Win32 module’s IsAdminUser() function), and if not invoke another copy of the script using something like runas.exe.

One last note in a similar vein: in several of the chapters I recommend using the Microsoft Scriptomatic tool to become familiar with WMI. By default this won’t run under Vista because it needs elevated privileges to function, but it is an “HTML Application” (.hta) file. Like Perl scripts, .hta files can’t easily be Run as administrator.

Here’s a recipe for getting around this limitation so you can use this excellent tool:

Right-click on the Internet Explorer icon in the taskbar (the “E”) and choose “Run as administrator” to run it using elevated privileges. (Warning: don’t use this running copy of IE to browse to any website or load anything but the Scriptomatic file, to be on the safe side.)
Press the Alt key to display the IE File menu. Choose “Open…” and then press the “Browse…” button. Change the dialog filter to display “All Files” and then browse to the location of the Scriptomatic .hta file. Open that file and you should be all set.

Locating and Installing Modules

Much of the benefit of using Perl for system administration work comes from all of the free code available in module form. The modules mentioned in this book can be found in one of three places:

The Comprehensive Perl Archive Network: CPAN is a huge archive of Perl source code, documentation, scripts, and modules that is replicated at over a hundred sites around the world. Information on CPAN can be found at http://www.cpan.org. The easiest way to find the modules in CPAN is to use the search engine at http://search.cpan.org. The “CPAN Search” box makes it simple to find the right modules for the job.
Individual repositories for prebuilt packages: In a moment we’ll encounter the Perl Package Manager (PPM), an especially important tool for Win32 Perl users. This tool connects to repositories (the most famous one is housed at ActiveState) to retrieve prebuilt module packages. A good list of these repositories can be found in the wiki at http://win32.perl.org. If a Win32 package we use comes from a repository other than ActiveState’s, I’ll be sure to point you to it.
Individual websites: Some modules are not published to CPAN or any of the PPM repositories. I really try to avoid them if possible, but in those rare cases where they fill a critical gap, I’ll tell you where to get them.

How do you install one of these modules when you find it? The answer depends on the operating system you are running. Perl now ships with documentation on this process in a file called perlmodinstall.pod (type perldoc perlmodinstall to read it). The next sections provide brief summaries of the steps required for each operating system used in this book.

Installing Modules on Unix

In most cases, the process goes like this:

Download the module and unpack it.
Run perl Makefile.PL to create the necessary Makefile.
Run make to build the package.
Run make test to run any test suites included with the module by the author.
Run make install to install it in the usual place for modules on your system.

If you want to save yourself the trouble of performing all these steps by hand, you can use the CPAN module by Andreas J. König (shipped with Perl), or the CPANPLUS module by Jos Boumans. CPAN allows you to perform all of those steps by typing:

% cpan
cpan[1]> install modulename

and CPANPLUS does the same with:

% cpanp
CPAN Terminal> i modulename

Both modules are smart enough to handle module dependencies (i.e., if one module requires another module to run, it will install both modules for you automatically). They also each have a built-in search function for finding related modules and packages. I recommend typing perldoc CPAN or perldoc CPANPLUS on your system to find out more about all of the handy features of these modules.

Installing Modules on Win32

The process for installing modules on Win32 platforms using the ActiveState distribution mirrors that for Unix, with one additional step: the Perl Package Manager (PPM). If you are comfortable installing modules by hand using the Unix instructions in the previous section, you can use a program like WinZip to unpack a distribution and use nmake instead of make to build and install a module.

Some modules require compilation of C files as part of their build process. A large portion of the Perl users in the Win32 world do not have the necessary software installed on their computers for this compilation, so ActiveState created PPM to handle prebuilt module distribution.

The PPM system is similar to that of the CPAN module. It uses a Perl program called ppm.pl to handle the download and installation of special archive files from PPM repositories. You can start the program either by typing ppm or by running ppm-shell from within the Perl bin directory:

C:\Perl\bin> ppm-shell
ppm 4.03
ppm> install module-name

PPM, like CPAN, can search the list of available and installed modules for you. Type help at the ppm> command prompt for more information on how to use these commands.

It’s Not Easy Being Omnipotent

Before we continue with the book, let’s take a few minutes for some cautionary words. Programs written for system administration have a twist that makes them different from most other programs: on Unix and Windows they are often run with elevated privileges (i.e., as root or Administrator). With this power comes responsibility. There is an extra onus on us as programmers to write secure code. We write code that can and will bypass the security restrictions placed on mere mortals. Tiny mistakes can lead to severe disruptions for our users or damage to key system files. And, if we are not careful, less “ethical” users may use flaws in our code for nefarious purposes. Here are some of the issues you should consider when you use Perl under these circumstances.

Don’t Do It

By all means, use Perl. But if you can, avoid having your code run in a privileged context. Most tasks do not require root or Administrator privileges.

For example, your log analysis program probably does not need to run as root. Create another, less privileged user for this sort of automation. Have a small, dedicated, privileged program hand the data to that user if necessary, and then perform the analysis as the unprivileged user.

Drop Your Privileges As Soon As Possible

Sometimes you can’t avoid running a script as root or Administrator. For instance, a mail delivery program you create may need to be able to write to a file as any user on the system. However, programs like these should shed their omnipotence as soon as possible during their run.

Perl programs running under Unix can set the $< and $> variables:

# permanently drops privs
($<,$>) = (getpwnam('nobody'),getpwnam('nobody'));

This sets the real and effective user IDs to nobody, which exists on most Unix/Linux systems as an underprivileged user (you can create the user yourself if need be). To be even more thorough, you may wish to use $( and $) to change the real and effective group IDs as well.

Windows does not have user IDs per se, but there are similar processes for dropping privileges, and you can use runas.exe to run processes as a different user.

Be Careful When Reading Data

When reading important data like configuration files, test for unsafe conditions first. For instance, you may wish to check that the file and all of the directories in its path are not writable (since that would make it possible for someone to tamper with them). There’s a good recipe for testing this in Chapter 8 of the Perl Cookbook , by Tom Christiansen and Nathan Torkington (O’Reilly).

The other concern is user input. Never trust that input from a user is palatable. Even if you explicitly print Please answer Y or N:, there is nothing to prevent the users from answering with 2,049 random characters (either out of malice or because they stepped away from the computer and a two-year-old came over to the keyboard instead).

User input can be the cause of even more subtle trouble. My favorite example is the “poison NULL byte” exploit reported in an article on Perl CGI problems (cited in the references section at the end of this chapter—be sure to read the whole article!). This particular exploit takes advantage of the difference between Perl’s handling of a NULL (\000) byte in a string and the handling done by the C libraries on a system. To Perl, there is nothing special about this character, but to the libraries it indicates the end of a string.

In practical terms, this means it is possible for a user to evade simple security tests. One example given in the article is that of a password-changing program whose code looks like this:

if ($user ne "root"){ <call the necessary C library routine> }

If a malicious user manages to set $user to root\000 (i.e., root followed by a NULL byte), the test will think that the name is not root and will allow the Perl script to continue. But when that string is passed to the underlying C library, the string will be treated as just root, and the user will have walked right past the security check. If not caught, this same exploit will allow access to random files and other resources on the system. The easiest way to avoid being caught by this exploit is to sanitize your input with something like this:

$input =~ tr/\000//d;

or better yet, only use valid data that you’ve explicitly extracted from the user’s input (e.g., with a regular expression).

Note

This is just one example of how user input can get programs into trouble. Because user input can be so problematic, Perl has a security precaution called taint mode. See the perlsec manpage that ships with Perl for an excellent discussion of “taintedness” and other security precautions.

Be Careful When Writing Data

If your program can write or append to every single file on the local filesystem, you need to take special care with how, where, and when it writes data. On Unix systems, this is especially important because symbolic links make file switching and redirection easy. Unless your program is diligent, it may find itself writing to the wrong file or device. There are two classes of programs where this concern comes especially into play.

Programs that append data to a file fall into the first class. The steps your program should take before appending to a file are:

Check the file’s attributes before opening it, using stat() and the normal file test operators. Make sure that it is not a hard or soft link, that it has the appropriate permissions and ownership, etc.
Open the file for appending.
stat() the open filehandle.
Compare the values from steps 1 and 3 to be sure that you have an open handle to the file you intended.

The bigbuffy program in Chapter 10 illustrates this procedure.

Programs that use temporary files or directories are in the second class. Chances are you’ve often seen code like this:

open(TEMPFILE,">/tmp/temp.$$") or die "unable to write /tmp/temp.$$:$!\n";

Unfortunately, that’s not sufficiently secure on a multiuser machine. The process ID ($$) sequence on most machines is easily predictable, which means the next temporary filename your script will use is equally predictable. If others can predict that name they may be able to get there first, and that’s usually bad news.

The easiest way to avoid this conundrum is to use Tim Jenness’s File::Temp module, which has shipped with Perl since version 5.6. Here’s how it is used:

use File::Temp qw(tempfile);

# returns both an open filehandle and the name of that file
my ($fh, $filename) = tempfile();
print $fh "Writing to the temp file now...\n";

File::Temp can also remove the temporary file for you automatically if desired. See the module’s documentation for more details.

Avoid Race Conditions

Whenever possible, avoid writing code that is susceptible to race condition exploits. The traditional race condition starts with the assumption that the following sequence is valid:

Your program will amass some data.
Your program can then act on that data.

Here’s a simple example:

Your program checks the timestamp on a file of bug submissions to make sure nothing has been added since you last read the file.
Your program modifies the contents of the file.

If users can break into this sequence at a point we’ll call “step 1.5” and make some key substitutions, they may cause trouble. If they can get your program in step 2 to naively act upon different data from what it found in step 1, they have effectively exploited a race condition (i.e., their program won the race to get at the data in question). Other race conditions occur if you do not handle file locking properly.

Race conditions often show up in system administration programs that scan the filesystem as a first pass and then change things in a second pass. Nefarious users may be able to make changes to the filesystem right after the scanner pass so that changes are made to the wrong file. Make sure your code does not leave such gaps open.

Enjoy

It is important to remember that system administration is fun. Not all the time, and not when you have to deal with the most frustrating of problems, but there’s definitely enjoyment to be found. There is a real pleasure in supporting other people and building the infrastructures that make users’ lives better. When the collection of Perl programs you’ve just written brings other people together for a common purpose, there is joy.

So, now that you’re ready, let’s get to work on those wires.

References for More Information

http://www.dwheeler.com/secure-programs/ is a HOWTO document written by David A. Wheeler for secure programming under Linux and Unix. The concepts and techniques Wheeler describes are applicable to other situations as well.

http://nob.cs.ucdavis.edu/bishop/secprog/ contains more good secure programming resources from security expert Matt Bishop.

http://www.homeport.org/~adam/review.html lists security code review guidelines by Adam Shostack.

http://www.canonical.org/~kragen/security-holes.html is an old but good paper on how to find security holes (especially in your own code) by Kragen Sitaker.

“Perl CGI Problems,” by rain.forest.puppy (Phrack Magazine, 1999), describes CGI security vulnerabilities. It can be found online at http://www.insecure.org/news/P55-07.txt or in the Phrack archives at http://www.phrack.com/issues.html?issue=55.

Perl Cookbook, Second Edition, by Tom Christiansen and Nathan Torkington (O’Reilly), contains many good tips on coding securely.

Automating System Administration with Perl, 2nd Edition by David N. Blank-Edelman

Chapter 1. Introduction