Managing the Testing Process: Practical Tools and Techniques for Managing Hardware and Software Testing

7.6. Keeping the Test Environment Clean

As I've stressed before, it is important to know precisely what is on each test system at any given time during test execution. Otherwise, bugs can become irreproducible and bug reports will be inaccurate. Although this might sound obvious, many people underestimate the importance of strict configuration management in test environments.

In my experience, the fastest way to let your test environment deteriorate into a mysterious state is to allow someone with a "hacker mentality" to work on your test systems. Just one letter altered in one configuration file, one cable disconnected, one software package installed can change the whole demeanor of a bug. Unless everyone working on your test system logs all changes to every test platform, your test environment will quickly become uncertain. The frustration engendered by bugs that change shape daily has made me a firm believer in scrupulous configuration management.

Most developers want to be able to make changes and try quick experiments when they're debugging; having to document every single step they try slows them down. Unfortunately, this often puts the goal of preserving test platform configuration in conflict with the goal of rapidly debugging a problem. From the test perspective, the best possible solution to this problem lies in preventing any debugging activities from taking place on the test platforms. In a perfect world, the development team has its own development labs—or at least ...