Network Flow Analysis

Network Flow Analysis

by Michael W. Lucas
Released June 2010
Publisher(s): No Starch Press
ISBN: 9781593272036

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Buy on ebooks.com
Start your free trial

Book description

Network flow analysis is the art of studying the traffic on a computer network. Understanding the ways to export flow and collect and analyze data separates good network administrators from great ones. The detailed instructions in Network Flow Analysis teach the busy network administrator how to build every component of a flow-based network awareness system and how network analysis and auditing can help address problems and improve network reliability.

Table of contents

  1. Network Flow Analysis
    2. ACKNOWLEDGMENTS
    3. INTRODUCTION
      1. Network Administration and Network Management
      2. Network Management Tools
        1. MRTG, Cricket, and Cacti
        2. RTG
        3. Nagios and Big Brother
        4. CiscoWorks, OpenView, and More
      3. Enough Griping: What's the Solution?
      4. Flow-Tools and Its Prerequisites
      5. Flows and This Book
    4. 1. FLOW FUNDAMENTALS
      1. What Is a Flow?
      2. Flow System Architecture
      3. The History of Network Flow
        1. NetFlow Versions
          1. NetFlow Version 1
          2. NetFlow Version 5
          3. NetFlow Version 7
          4. NetFlow Version 8
          5. NetFlow Version 9
        2. NetFlow Competition
        3. The Latest Standards
      4. Flows in the Real World
        1. ICMP Flows
        2. UDP Flows
        3. TCP Flows
        4. Other Protocols
      5. Flow Export and Timeouts
      6. Packet-Sampled Flows
    5. 2. COLLECTORS AND SENSORS
      1. Collector Considerations
        1. Operating System
        2. System Resources
      2. Sensor Considerations
        1. Location
          1. Internet Border
          2. Ethernet Core
        2. From Remote Facilities
        3. From Private Network Segments/DMZs
      3. Implementing the Collector
      4. Installing Flow-tools
        1. Installing from Packages
        2. Installing from Source
      5. Running flow-capture
        1. Starting flow-capture at Boot
      6. How Many Collectors?
      7. Collector Log Files
      8. Collector Troubleshooting
      9. Configuring Hardware Flow Sensors
        1. Cisco Routers
        2. Cisco Switches
        3. Juniper Routers
      10. Configuring Software Flow Sensors
        1. Setting Up Sensor Server Hardware
        2. Network Setup
        3. Sensor Server Setup
        4. Running the Sensor on the Collector
      11. The Sensor: softflowd
        1. Running softflowd
        2. Watching softflowd
          1. Viewing Tracked Flows
          2. Viewing Flow Statistics
    6. 3. VIEWING FLOWS
      1. Using flow-print
        1. Printing Protocol and Port Names
        2. Common Protocol and Port Number Assignments
        3. Viewing Flow Record Header Information with -p
        4. Printing to a Wide Terminal
      2. Setting flow-print Formats with -f
        1. Showing Interfaces and Ports in Hex with Format -f 0
        2. Two Lines with Times, Flags, and Hex Ports Using -f 1
        3. Printing BGP Information
        4. Wide-Screen Display
        5. IP Accounting Format
      3. TCP Control Bits and Flow Records
      4. ICMP Types and Codes and Flow Records
        1. Types and Codes in ICMP
        2. Flows and ICMP Details
    7. 4. FILTERING FLOWS
      1. Filter Fundamentals
        1. Common Primitives
        2. Creating a Simple Filter with Conditions and Primitives
        3. Using Your Filter
      2. Useful Primitives
        1. Protocol, Port, and Control Bit Primitives
          1. IP Protocol Primitives
          2. Port Number Primitives
          3. TCP Control Bit Primitives
          4. ICMP Type and Code Primitives
        2. IP Address and Subnet Primitives
          1. IP Addresses
          2. Subnet Primitives
        3. Time, Counter, and Double Primitives
          1. Comparison Operators in Primitives
          2. Time Primitives
          3. Counter Primitives
          4. Double Primitives
        4. Interface and BGP Primitives
          1. Identifying Interface Numbers Using SNMP
          2. Interface Number Primitive
          3. Autonomous System Primitives
      3. Filter Match Statements
        1. Protocols, Ports, and Control Bits
          1. Network Protocol Filters
          2. Source or Destination Port Filters
          3. TCP Control Bit Filters
          4. ICMP Type and Code Filters
        2. Addresses and Subnets
        3. Filtering by Sensor or Exporter
        4. Time Filters
        5. Clipping Levels
          1. Octets, Packets, and Duration Filters
          2. Packets or Bits per Second Filters
        6. BGP and Routing Filters
          1. Autonomous System Number Filters
          2. Next-Hop Address Filters
          3. Interface Filters
      4. Using Multiple Filters
      5. Logical Operators in Filter Definitions
        1. Logical "or"
        2. Filter Inversion
      6. Filters and Variables
        1. Using Variable-Driven Filters
        2. Defining Your Own Variable-Driven Filters
        3. Creating Your Own Variables
    8. 5. REPORTING AND FOLLOW-UP ANALYSIS
      1. Default Report
        1. Timing and Totals
        2. Packet Size Distribution
        3. Packets per Flow
        4. Octets in Each Flow
        5. Flow Time Distribution
      2. Modifying the Default Report
        1. Using Variables: Report Type
        2. Using Variables: SORT
      3. Analyzing Individual Flows from Reports
      4. Other Report Customizations
        1. Choosing Fields
        2. Displaying Headers, Hostnames, and Percentages
        3. Presenting Reports in HTML
      5. Useful Report Types
        1. IP Address Reports
          1. Highest Data Exchange: ip-address
          2. Flows by Recipient: ip-destination-address
          3. Most Connected Source: ip-source-address-destination-count
          4. Most Connected Destination: ip-destination-address-source-count
        2. Network Protocol and Port Reports
          1. Ports Used: ip-port
          2. Flow Origination: ip-source-port
          3. Flow Termination: ip-destination-port
          4. Individual Connections: ip-source/destination-port
          5. Network Protocols: ip-protocol
        3. Traffic Size Reports
          1. Packet Size: packet-size
          2. Bytes per Flow: octets
          3. Packets per Flow: packets
        4. Traffic Speed Reports
          1. Counting Packets: pps
          2. Traffic at a Given Time: linear-interpolated-flows-octets-packets
        5. Routing, Interfaces, and Next Hops
          1. Interfaces and Flow Data
          2. The First Interface: input-interface
          3. The Last Interface: output-interface
          4. The Throughput Matrix: input/output-interface
          5. The Next Address: ip-next-hop-address
          6. Where Traffic Comes from and How It Gets There: ip-source-address/output-interface
          7. Where Traffic Goes, and How It Gets There: ip-destination-address/input-interface
          8. Other Address and Interface Reports
        6. Reporting Sensor Output
        7. BGP Reports
          1. Using AS Information
          2. Traffic's Network of Origin: source-as
          3. Destination Network: destination-as
          4. BGP Reports and Friendly Names
      6. Customizing Reports
        1. Custom Report: Reset-Only Flows
          1. Report Format and Output
          2. Removing Columns
          3. Applying Filters to Reports
          4. Combining stat-reports and stat-definitions
        2. More Report Customizations
          1. Reversing Sampling
          2. Filters in stat-report Statements
          3. Reporting by BGP Routing
        3. Customizing Report Appearance
          1. flow-rptfmt Options
          2. Dump CSV to a File
          3. Using Time to Direct Output
          4. Set Sorting Order
          5. Cropping Output
          6. Other Output Options
          7. Alternate Configuration Files
    9. 6. PERL, FLOWSCAN, AND CFLOW.PM
      1. Installing Cflow.pm
        1. Testing Cflow.pm
        2. Install from Operating System Package
        3. Install from Source
        4. Installing from Source with a Big Hammer
      2. flowdumper and Full Flow Information
      3. FlowScan and CUFlow
      4. FlowScan Prerequisites
      5. Installing FlowScan and CUFlow
        1. FlowScan User, Group, and Data Directories
        2. FlowScan Startup Script
        3. Configuring FlowScan
        4. Configuring CUFlow: CUFlow.cf
          1. Subnet
          2. Network
          3. OutputDir
          4. Scoreboard
          5. AggregateScore
          6. Router
          7. Service
          8. Protocol
          9. AS
        5. Rotation Programs and flow-capture
        6. Running FlowScan
        7. FlowScan File Handling
        8. Displaying CUFlow Graphs
      6. Flow Record Splitting and CUFlow
        1. Splitting Flows
        2. Scripting Flow Record Splitting
        3. Filtered CUFlow and Directory Setup
      7. Using Cflow.pm
        1. A Sample Cflow.pm Script
        2. Cflow.pm Variables
        3. Other Cflow.pm Exports
        4. Acting on Every File
        5. Return Value
        6. Verbose Mode
    10. 7. FLOWVIEWER
      1. FlowTracker and FlowGrapher vs. CUFlow
      2. FlowViewer Security
      3. Installing FlowViewer
        1. Prerequisites
        2. FlowViewer Installation Process
      4. Configuring FlowViewer
        1. Directories and Site Paths
        2. Website Setup
        3. Devices and Exporters
          1. One Collector per Sensor
          2. One Collector for All Sensors
        4. Troubleshooting the FlowViewer Suite
      5. Using FlowViewer
        1. Filtering Flows with FlowViewer
          1. Device
          2. Next Hop IP
          3. Start and End Date and Time
          4. TOS Field, TCP Flag, and Protocol
          5. Source and Dest IP
          6. Source and Dest Interface
          7. Source and Dest Port and AS
        2. Reporting Parameters
          1. Include Flow If
          2. Sort Field, Resolve Addresses, and Oct Conv, and Sampling Multip
          3. Pie Charts
          4. Cutoffs
        3. Printed Reports
        4. Statistics Reports
      6. FlowGrapher
        1. FlowGrapher Settings
          1. Detail Lines
          2. Graph Width
          3. Sample Time
          4. Graph Type
        2. FlowGrapher Output
      7. FlowTracker
        1. FlowTracker Processes
        2. FlowTracker Settings
          1. Tracking Set Label
          2. Tracking Type
          3. Sampling Multiplier
          4. Alert Threshold
          5. Alert Frequency
          6. Alert Destination
          7. General Comment
        3. Viewing Trackers
        4. Group Trackers
      8. Interface Names and FlowViewer
    11. 8. AD HOC FLOW VISUALIZATION
      1. gnuplot 101
        1. Starting gnuplot
        2. gnuplot Configuration Files
      2. Time-Series Example: Bandwidth
        1. Total Bandwidth Report
          1. Filtering Flows for Total Traffic
          2. The Target Graph
          3. The First Graph: Missing the Target
          4. Changing How the Graph Is Drawn
          5. Clipping Levels
          6. Printing Graphs to Files
          7. Save Your Work!
        2. Unidirectional Bandwidth Reports
          1. Filtering Flows for Unidirectional Traffic
          2. Creating a Unidirectional Graph
        3. Combined Inbound/Outbound Traffic
          1. Preparing the Data Files
          2. Displaying Two Graphs Simultaneously
      3. Automating Graph Production
      4. Comparison Graphs
        1. Data Normalizing
        2. Time Scale
    12. 9. EDGES AND ANALYSIS
      1. NetFlow v9
        1. Installing flowd
        2. Configuring flowd
        3. Converting flowd Data to Flow-tools
      2. sFlow
        1. Configuring sFlow Export with sflowenable
        2. Convert sFlow to NetFlow
      3. Problem Solving with Flow Data
        1. Finding Busted Software
          1. Broken Connection Filters
          2. Checking for Resets
          3. Checking for Failed Connections
        2. Identifying Worms
        3. Traffic to Illegal Addresses
        4. Traffic to Nonexistent Hosts
      4. Afterword
    13. About the Author
    14. UPDATES

Product information

  • Title: Network Flow Analysis
  • Author(s): Michael W. Lucas
  • Release date: June 2010
  • Publisher(s): No Starch Press
  • ISBN: 9781593272036