The Reign of Botnets

5Assessing Detection Accuracy

Bot detection is about classifying web traffic as human or bot. Regular assessment of the traffic is necessary to ensure optimal user experience and prevent bot traffic from accessing the web resource. This chapter provides a methodology on how to assess the accuracy of the detection visually.

Prerequisites

Detecting bots, fraud, and abuse online is not a perfect science. Despite every effort vendors make to create products that are as accurate as possible, results may vary due to the ever-changing threat landscape and evolution of the Internet digital ecosystem. To ensure the optimal performance of the solution to keep the malicious traffic at bay while providing legitimate users with the best experience possible, regular assessment of the traffic is necessary to adjust the detection methods when necessary.

This chapter describes a proven method for assessing any detection method. It assumes you know and understand the OSI model and the TCP/IP, TLS, and HTTP protocols. A strong knowledge of the HTTP protocol and, in particular, the meaning of the various headers, as described in the “HTTP Headers 101” section in Chapter 3, “The Evolution of Botnet Attacks,” is essential. You must also have an adequate understanding of how websites are structured and a good understanding of JavaScript and how it may trigger API calls or Ajax requests to retrieve ...