Chapter 8. Parsing Web Access Logs
Web server access logs are an excellent source of information about what your site’s visitors are up to. The information on separate visitors is all mixed together, though, and for all but the smallest sites the raw access logs are too large to read directly. What you need is log analysis software to make the information in the log more easily accessible. You can buy commercial log analysis software to do this, but Perl makes it easy to write your own. The next three chapters describe how to build such a home-grown log-analysis tool.
This chapter focuses on the first part of the process: extracting and storing the information we’re interested in. We talk about log file structure, converting IP addresses, and creating regular expressions capable of parsing web access logs. We also talk about creating a suitable data structure for storing the extracted data, so we can answer interesting questions about what our site’s visitors have been doing. Along the way we discuss the difficulty of identifying those visitors in the web server’s log entries and devise an approach for extracting at least an approximate version of that information.
The example continues in Chapter 9, which focuses on how to do computations involving dates and times, and finishes in Chapter 10, which covers the specifics of how we manipulate the “visit” information from our logs, as well as the actual output of the finished report.
Log File Structure
Most web servers store their ...
Get Perl for Web Site Management now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.