Modern System Administration

Appendix B. Resolving Test Failures

Building on Chapter 7, let’s learn more about the different types of test failures (environmental problems, flawed test logic, changing assumptions, flaky tests, and code defects).

Test Failure Type #1: Environment Problems

Environmental problems can be super frustrating because so much can go wrong, especially in the larger end-to-end tests that test between different services. Make sure to have sufficient unit test coverage because unit tests are not vulnerable to environmental problems. There are many issues with environments that can arise, including these:

• The testing environment doesn’t match the production environment in scale or function.

• Elements of functionality can be costly, for example, monitoring agents that shouldn’t have an impact but do.

• There is no local testing environment setup due to lack of understanding that it’s possible to have a local testing environment.

• Dependencies aren’t locked down and vary in environments.

• Third-party continuous integration and deployment services are having failures.

These are just a few examples of environmental problems that can cause tests to fail.

Problems with shared test environments can lead to folks insisting that no testing environment is needed, and they instead test directly in production with feature flags and canary testing. Feature flags make features available to a subset of users in a controlled manner or turn off a feature if necessary. Canary testing ...