Cisco IOS Access Lists By Jeff Sedayao June 2001 1-56592-385-5, Order Number: 3855 272 pages, $39.95

Chapter 5
Debugging Access Lists

In this chapter:
Router resource access control lists
Packet-filtering access control lists
Route-filtering access control lists

Once you've formatted access lists and used them to implement policies, how do you know if your access lists are correct? How can you find problems with them? We'll look at these questions in this chapter, first verifying that your access lists are working correctly in the areas of router resource control, packet filtering, and route filtering. More generally, I will talk about how access lists can go wrong and what are the typical failure modes of access lists. Finally, we'll look at some tips and tricks for debugging access lists in detail.

Router resource access control lists

In this section, I discuss how to debug router resource access lists. The first part describes how to check them for correctness since it doesn't make sense to debug a list that is configured properly. The second part discusses what generally happens when access lists go wrong, and the last part goes over specifically how to debug router resource access lists.

Checking for correctness

In Chapter 3 we configured the router to control resources such as Telnet and time services. The approach to verifying if these access lists function correctly is very basic: test if access works correctly for those who are permitted, and test if access does not work for those who are not permitted. Let's look at one of our early examples of router resource policies and look at how we can test it. In the first example in Chapter 2, we had a policy like the following:

Only the hosts at IP addresses 192.168.30.1 and 192.168.33.5 may telnet into the router

The access list that defines the policy set for this policy is:

access-list 1 permit 192.168.30.1
access-list 1 permit 192.168.33.5

We use the policy set as follows:

line vty 0 4
 access-class 1 in

In order to verify that the access list actually implements the policy, we need to check that what we defined in the policy set matches what is in the policy definition. There are two ways to do this. The first is by inspection: we manually check whether the access list matches our policy. While this method can work for small access lists, it becomes much more difficult as access lists grow in size and really isn't a particularly reliable way to verify that an access list is correct. The second way is to test whether access lists actually function as desired. In this instance, we would attempt to telnet to the router from the hosts at 192.168.30.1 and 192.168.33.5. If we succeed in getting a prompt from the router, we know that the access list is allowing the correct host to connect to the router.

We should also make sure that forbidden hosts do not have access. If for some reason we forgot to apply an access list (not putting in the access-class statement, for example), the default policy set permits everything, giving us the same results as the test we did earlier. When we telnet to the router from the host at 192.168.3.59, the router should refuse the connection and not provide a login prompt. In general, the following algorithm is useful for verifying a policy against the access lists you implemented:

Make sure what hosts that are permitted have access to the resource

Make sure that the hosts that are not permitted do not have access to the resource

You will not always be able to completely test all cases (as access lists grow, you will not be able to test every entry), but being reasonably sure that the above two conditions are true is how to verify correctness.

Manual tests of masks

Testing for both what is permitted and what is denied is particularly useful when you use masks in a policy. Let's look at the following policy and its implementation:

Only the hosts at IP addresses 192.168.30.4 through 192.168.30.7 and IP address 192.168.33.5 may telnet into the router

We define the appropriate policy set and apply it as follows:

access-list 1 permit 192.168.30.4 0.0.0.3
access-list 1 permit 192.168.33.5
! line definition
line vty 0 4
access-class 1 in

Notice that the first access list entry has a mask, as we covered in Chapter 2. When you use a mask like this, in addition to testing that the hosts in the mask range have access to the resource being controlled, make sure that the hosts just outside of the mask range (hosts at IP addresses 192.168.30.3 and 192.168.30.8) do not. What happens if you specify a mask that is too large, as in the following list?

access-list 1 permit 192.168.30.4 0.0.0.7
access-list 1 permit 192.168.33.5

Testing only the permitted hosts will miss the fact that you also included hosts 192.168.30.1, 192.168.30.2, and 192.168.30.3 in the policy set. Testing the hosts just outside the range permitted by the mask catches this problem.

When access lists don't work

I have talked about making sure that access lists are functioning properly by implementing the policies that you intend. But what happens if your access lists do not function as expected? This section describes how an access list can do other than what you intend and how you can use the various tools available on a Cisco router to find where you made a mistake. I will first go over ways that access lists can go wrong. After that, I will cover how to debug router resource access lists, extended access lists, and router filtering access lists.

There are typically a number of ways access lists can go wrong. First, they can be applied incorrectly, meaning that you have either applied the wrong access list to a router resource, interface, or distribute list; applied the correct access list but with the wrong directionality (inbound instead of outbound or the reverse); or forgotten to apply one altogether. This should be the first thing you check.

If you are applying an access list correctly, and your policy is still not being implemented properly, then one of two things is happening. Either something you want to include into a policy set was excluded, or something you want excluded from a policy set was included. In the first case, you need to check whether some statement in the access list is excluding the IP addresses or packet types that you want in the policy set. (If you are certain that nothing is excluding the items desired, then the only other explanation is that you have forgotten to include it.) In the second case, you have included something in your policy set that you do not want, so to fix your access list, you need to find the permissive entry. These are the fundamental problems that can occur, and when I talk about debugging specific kinds of access lists in the following sections, I go over how to find bad or missing access list entries.

There is one final category of access list problems that you may encounter. If you are implementing security policies and routing policies, you have to be careful about their interaction. For instance, if an application does not work through an extended access list, it doesn't always mean there's a problem with the access lists; it could be a communication problem between the two systems you are routing. If you encounter what seems to be a routing problem with routing, there could be packet filtering that is disrupting route advertisements. I will also talk about this category of problems in later sections on debugging.

Debugging router resource access lists

If you find that your router resource access lists are not working, typically one of two things is happening. Either something that needs access to a router resource does not have access, or something that should not have access to the resource does. I'll go over each case and what to look for when trying to find what the problem is.

If a host or router that should have access cannot access a resource on a router you control, the first thing to check is whether there is network connectivity between the host in question and the router. The easiest way to do this is to use the ping command. The format of ping is the command ping followed by the name or IP address of the host you want to check on. Let's say that we want to check on connectivity to a host at 10.1.1.2, and the route to 10.1.1.2 goes through an Ethernet interface with IP address 192.168.3.2. We would use the following command, which can be executed from user EXEC mode or privileged EXEC mode:

ping 10.1.1.2

If there is a functioning route to IP address 10.1.1.2 and a route from the host 10.1.1.2 back to the router interface with IP address 192.168.3.2, we would see output like this:

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

If either the transmit and return path between 10.1.1.2 and 192.168.3.2 is unavailable, the ping will not be successful:

Router1# ping 10.1.1.2
 
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

If the ping attempt is not successful, there is either a route missing at the target host for the router, a route missing at the router for the host, or some other packet filter along the path getting in the way. Let's put off a discussion of packet filtering extended access lists until the next section and assume that the problem is with a missing route. Check the routing on the host or the router. If the route is missing, and there is no default route, then you need to make sure that the route is there and again check if the resource access works. If the route is there, it implies that the problem may be with a packet filter along the way. I will talk about finding problematic packet filters in later sections.

If the ping attempt is successful, the problem is with your access list. You either excluded some IP addresses from your policy set or forgot to include an appropriate entry.

Looking at an example will illustrate this better. Let's say we have the following policy for a router:

All of the hosts with IP addresses 192.168.32.32 through 192.168.32.63 except 192.168.32.40 through 192.168.32.43 can have SNMP read access

and implement it as follows:

access-list 1 deny 192.168.40.0 0.0.0.7
access-list 1 permit 192.168.32.0 0.0.0.31
snmp community string public ro 1

We notice that host 192.168.32.44 does not have SNMP access. We can ping 192.168.32.44 from the router, so routing is not an issue. On examination, we see that the initial deny mask is too large. Access list 1 should be defined as:

access-list 1 deny 192.168.40.0 0.0.0.3
access-list 1 permit 192.168.32.0 0.0.0.31

The other way that router resource access lists can go wrong is if something that should not have permission does have permission. This means that somehow you have inadvertently allowed something in a policy set you shouldn't have, and again a previous example illustrates this well. Recall that we have a policy for a router as follows:

Only the hosts at IP addresses 192.168.30.4 through 192.168.30.7 and IP address 192.168.33.5 may telnet into the router

We define the appropriate policy set and apply it as follows:

access-list 1 permit 192.168.30.4 0.0.0.7
access-list 1 permit 192.168.33.5
! line definition
line vty 0 4
access-class 1 in

After testing, we notice that host 192.168.30.8 has Telnet access to the router. We made the mask on the first entry of access list 1 too large. Access list 1 should be:

access-list 1 permit 192.168.30.4 0.0.0.3
access-list 1 permit 192.168.33.5

Packet-filtering access control lists

Here I talk about debugging the packet filters that you implement with access control lists. Like the previous section, I first talk about how to verify that your access lists are correct, followed by a section about how to find the problems in the access lists that you find to be wrong.

Checking for correctness

One of the first things you want to do is make sure that your access lists are applied to the interfaces you intended. You or another network administrator may have removed access lists or applied other access lists in order to debug problems or temporarily enable certain functionality for a variety of reasons, such as host installations or debugging. One way to do that is to show the running configuration with the show running-confg command. If you have a large configuration, this command may take a while, and it is easy to miss the interface you want to look at when many of them are scrolling by.

Using show ip interface to display applied access lists

A better way is to use the show ip interface command. This command yields output that looks like the following:

Serial 0 is up, line protocol is up 
 Internet address is 192.168.1.2/24 
 Broadcast address is 192.168.1.255 
 Address determined by non-volatile memory 
 MTU is 1500 bytes 
 Helper address is not set 
 Directed broadcast forwarding is enabled
 Outbound access list is 102 
 Inbound access list is 101 
 Proxy ARP is enabled 
 Security level is default 
 Split horizon is enabled 
 ICMP redirects are always sent 
 ICMP unreachables are always sent 
 ICMP mask replies are never sent 
 IP fast switching is enabled 
 IP fast switching on the same interface is enabled 
 IP Optimum switching is disabled IP 
 Flow switching is enabled IP 
 CEF switching is enabled IP 
 Distributed switching is enabled 
 IP LES Flow switching turbo vector IP 
 Flow CEF switching turbo vector 
 IP multicast fast switching is disabled 
 IP multicast distributed fast switching is disabled 
 Router Discovery is disabled 
 IP output packet accounting is enabled 
 IP access violation accounting is disabled 
 TCP/IP header compression is disabled 
 Probe proxy name replies are disabled 
 Gateway Discovery is disabled 
 Policy routing is disabled 
 Web Cache Redirect is disabled 
 BGP Policy Mapping is enabled (source ip-prec-map)

The show ip interface command displays what IP configurations are applied relative to an interface. On this serial interface, we can see that it has an Inbound access list of 101 and an Outbound access list of 102. Further down the listing, we can see that IP output packet accounting is enabled, while IP access violation accounting is disabled. Though I'll talk about these last two features later, you can see that this command shows the access lists applied to a particular interface, allowing you to verify that you used access lists correctly.

The show ip interface command followed by an interface shows the IP configuration for just that interface. Without the interface specification, the command shows the IP configuration for all interfaces.

Testing the functionality of packet filters

Once you know that your access lists are properly applied, the most direct way of checking packet filtering access lists is the same method as for router resource access lists: test if what you permit is allowed and what is denied is not allowed through your router. Let's look at some of the extended access lists to see how we can verify access list correctness. Recall our first example of extended access lists, which implemented the following policy:

HTTP and SSL packets only to the host at 192.168.35.1

We created a policy using the following access list:

access-list 101 permit any host 192.168.35.1 eq http
access-list 101 permit any host 192.168.35.1 eq 443

and applied it with:

interface ethernet 0
access-group 101 out

The host with IP address 192.168.35.1 lies on Ethernet 0.

How do you verify that the access list fulfills the policy? As with verifying access lists for router resource restriction, you can manually check if the applications that use these protocols are working. Since the default access list permits everything (recall that an access list that is applied but has no entries permits everything), you also need to check that blocked applications do not work. For this example, you might start the web server on host 192.168.35.1, open a web browser on a host on a different segment, and then see if you can access the web server and do SSL transactions. If web access and SSL transactions work, you then check to see if other applications do not work. An easy way to test this is to make sure that a Telnet connection attempt to the host times out. Telnet is particularly convenient because Telnet clients are easily available. You would type the following on the test host:

telnet 192.168.35.1

If the access list functions correctly, the connection attempt should time out. If you do manage to connect or receive a connection refused message (meaning that the web server does not run Telnet service on the standard Telnet port), the access list is not functioning as intended.

TCP port probing using Telnet

As a network administrator, you will not always have the luxury of testing the applications that you permit or deny through a router. In that case, there are a number of ways to check that your access lists are working correctly. These methods are not as reliable as actually testing the applications--they cannot show that your access list is absolutely right--but they can let you know if you did something wrong. As an example of this, let's look at how we can verify the correctness of our current example if we don't have access to a web browser. The network is shown in Figure 5-1.

Figure 5-1. Checking extended access list correctness without application access

After implementing access list 101 on Ethernet 0 of Router 1, we need to verify that the list functions correctly--meaning it implements our policy, assuming host 192.168.33.1 lacks a web browser and we only have access to Router 2, which also lacks a web browser. How can we verify correctness in either scenario?

One simple technique you can use is port probing. The telnet command makes it easy to probe if TCP-based services are accessible. From either the host at 192.168.33.1 or Router 2, we can see which ports on the web server are accessible through Router 1. If the access list is working correctly, when we execute the following commands on either the host or the router:

telnet 192.168.35.1 80
telnet 192.168.35.1 443

we should get a connection setup confirmation or a connection refused message.

The first command attempts to connect to the web server's HTTP port. The second command attempts to connect to the web server's SSL port. Since access list 101 allows any host to access these two ports on the web server, connecting to either port should evoke a response from the web server. If the connection attempt times out, and we know that the web server is operational, then our access list must be incorrect. As in previous examples, we also need to check whether the access list is blocking other protocols with:

telnet 192.168.35.1 

The result of this particular telnet command should be a timeout; in other words, connections should not be achieved or refused.

You can specify which interface you want to telnet from a router using the global configuration command ip telnet source-interface followed by an interface. This command is useful when you want to test the path through a particular interface of a router.

Using telnet to probe ports is convenient. You can use any host or router to run tests like this. A network administrator implementing access lists is much more likely to have access to a router than to a host running the permitted application, since the administrator often has access only to network equipment while hosts are commonly run by other organizations. The Telnet technique does have its limitations. It can be time-consuming to check large numbers of ports.

TIP:   Only TCP ports can be probed in this manner. To check both UDP and TCP port availability, port scanning software is available. These scanning packages check every UDP and TCP port that is accessible on a network that you can define. While automated and much more thorough than manual port checks, this software typically can only be run from a host and is generally not available on routers.

Access list entry accounting

Another way to verify if your access lists are correct is to see which access list entries are being used. When an application sends traffic through a router and that traffic gets filtered through an access list, then the Cisco router doing the packet filtering logs which entries of the access list are used. The show access-list command not only shows the entries an access list has, but also how many times a particular access list entry has been used. For our simple example, the command show access-list 101 should yield something like this:

access-list 101
      permit any host 192.168.35.1 eq www (10 matches)
      permit any host 192.168.35.1 eq 443 (1 match)

An access list entry that functions properly should generate matches when users use the application that the entry tries to control. If there is successful www (web) access to the web server at 192.168.35.1, then the entry allowing web access to the server should have matches. If there are no matches when application traffic is generated, then there is something wrong with the access list.

This method is not sufficient to prove that all of a router's access lists are correct. For example, if we have an access list blocking traffic from the web server to certain clients (on Ethernet 1 in this case) but not others, we could have www matches to the web server on access list entries, but our policy could still not be implemented correctly. Though not a sufficient condition for verifying correctness, access list entry matches are a necessary condition for correctness and can be a useful indicator of problems.

IP accounting

The IP accounting facility is another way to check if extended access lists are correct. IP accounting lets you keep track of the source and destination IP addresses of transit packets (packets that go out of an interface and come in through an interface, possibly the same interface). It also allows you to track the source and destination IP addresses of packets that violate access lists--packets that are not accepted into the policy set of packets allowed through an interface. There are two ways you can use this capability. Transit packet IP accounting shows you what packets have successfully made it through an access list on a given interface, while access violation IP accounting tracks the packets that have been denied.

Let's turn on IP accounting for our web server example and see how we can use it. These tools are enabled with the following configuration fragment:

interface ethernet 0
 ip accounting output-packets
 ip accounting access-violations
interface ethernet 1
 ip accounting output-packets
 ip accounting access-violations

The ip accounting interface statement without a keyword turns on IP accounting for transit packets. Like outgoing access lists, IP accounting does not capture packets generated from the router, such as NTP queries or Telnet sessions originated from the router, so don't expect those types of traffic to be recorded by IP accounting. Once we have configured IP accounting for output packets and access violations, we need to look at the contents of the accounting database. We do this with the show ip accounting command. For our web server example, this yields:

Source           Destination        Packets       Bytes
192.168.31.2     192.168.35.1             9       13052
192.168.35.1     192.168.31.2            10      957200
192.168.35.1     192.168.70.2             6        8572
192.168.70.2     192.168.35.1             8        1303
 
Accounting data age is 10

Here you can see what traffic is going through the router, in particular, traffic going to and from the web server. Packets with a destination of 192.168.35.1 (the web server IP address) have made it through the access lists. Return traffic (with a source IP address of the web server) has also been recorded. Each accounting table entry shows the number of packets and bytes sent from one source IP address to a destination IP address. The final line tells you how many minutes have passed since the accounting table has been turned on or cleared. If you see traffic to and from some other host (perhaps 192.168.35.2), then you know you have a problem with your access lists.

To see if your access lists are rejecting what they should accept, use the show ip accounting access-violations command. For our example, it should yield this:

Source            Destination              Packets               Bytes ACL
192.168.75.2      192.168.35.2                  65               36943 101
192.168.30.1      192.168.35.3                  24               16115 101
192.168.152.26    192.168.35.1                   8                 818 101
 
Accounting data age is 3

You can see that attempts to reach host 192.168.35.2 fail because access to this host is not permitted. This is correct and functioning as we intended. What about the last line? Didn't we allow access to host 192.168.35.1? We did, but only for web and SSL, and we verified earlier that web and SSL traffic could get through. If someone tried to connect to 192.168.35.1 with a protocol other than SSL or HTTP, then this would also show up as an access violation.

Managing the IP accounting database One database is used for storing both transit accounting and access list violation information. This single database is limited in size, and accounting information is lost when the database is full and no new entries can be entered. You can see when the database is full if you see a message like: Accounting threshold exceeded for 13475 packets violating access list(s) at the end of show ip accounting output. There are several ways to deal with this. To make the table bigger, you can use the ip accounting-threshold configuration directive, followed by the size of the database you want. The default size is 512 entries. Keep in mind, though, that the accounting table takes up router memory. If memory becomes an issue, you may not be able to increase the size of the accounting table as much as you may like. While both the access list violation information and the transit information are stored in the same database, you can define how many entries are dedicated to each. The directive ip accounting transits, followed by the number of entries you want, lets you set the number of entries in the accounting table that are dedicated to storing IP packet transit info. You can limit what entries go into the IP accounting table using the ip accounting-list statement followed by an IP address and an optional mask. For example, the following configuration commands: ip accounting-list 192.168.30.1 ip accounting-list 10.0.0.0 0.255.255.255 allow only packets originating from or destined for host 192.168.30.1 or network 10.0.0.0/8 into the IP accounting database. Keep in mind, however, that this can be yet another access list that you may need to debug! Finally, another way to deal with a lot of accounting information is to periodically clear the database. The command clear ip accounting can be used to empty the IP accounting table. You also may want to use this command after making access list changes. Keep in mind that when using IP accounting, as with access list entry accounting, correct behavior as seen from this tool is a necessary but not sufficient condition of correct implementation. IP accounting does not provide any information about ports or protocols used, so the packets you see in the database may not be the ones you expect.

Debugging extended access lists

As with router resource access lists, extended access lists used for packet filtering have two failure modes: some application or utility that should be permitted is denied or something that should be denied is permitted. For the first case, as with router resource access lists, you need to verify routing in order to find the problem. However, using ping may or may not work this time because of the packet filters you are setting up. In any case, you have to ensure that the end points of an application, the client and server, have routes to each other. If you have verified the routing, and your application still does not work, you have to find where and how you inadvertently excluded packets from your target policy, which I'll describe shortly.

One common way to break applications with extended access lists is not to take into account packet flow in both directions: from client to server and from server to client. As an example, let's say we are trying to permit FTP from everywhere to an FTP server host set up as in Figure 5-2.

Figure 5-2. An FTP server

Since ports 20 and 21 are used on the server, let's set up our router as follows:

access-list 101 permit tcp any host 192.168.35.1 range 20 21
access-list 102 permit tcp host 192.168.35.1 any established
interface ethernet 0
access-group 101 out
access-group 102 in

FTP will not work with this configuration. Why? While FTP servers do use ports 20 and 21 during an FTP transfer, the data connection is usually initiated from the FTP server with port 20 as a source port. Since only packets from already established connections are permitted from the FTP servers, clients can set FTP control sessions but can't do any data transfers. The following access lists take into account the packet flow of FTP:

access-list 101 permit tcp any host 192.168.35.1 range 20 21
access-list 102 permit tcp host 192.168.35.1 range 20 21 any gt 1023

In addition to not setting up connections in the direction you think (as in the previous example), there are other common instances in which applications do not behave the way that you think they will. In particular, the application may demand different source or destination ports than those you anticipated in your access lists. One way to check on actual port usage is to watch how an application actually behaves with the netstat command, available on most Unix and Windows systems. netstat prints TCP connection information, including the port numbers used for the TCP connections currently running on the box. For example, if you want to examine the port behavior of the Simple Mail Transport Protocol (SMTP), you could run netstat on a mail server. On a host called host1, netstat would yield something like this:

Active Internet connections
Proto Recv-Q Send-Q  Local Address         Foreign Address        (state)
tcp        0      0  host1.smtp            host2.1042             ESTABLISHED
tcp        0      0  host1.smtp            host3.4374             ESTABLISHED
tcp        0      0  host1.smtp            host4.1301             ESTABLISHED
tcp        0      0  host1.1252            host4.smtp             SYN_SENT
tcp        0      0  host1.smtp            host5.1249             ESTABLISHED
tcp        0      0  host1.1260            host4.smtp             ESTABLISHED
tcp        0      0  host1.smtp            host6.37688            ESTABLISHED
tcp        0      0  host1.1242            host7.smtp             SYN_SENT

The first column is the protocol. (In this excerpt, only the TCP connections are shown.) The next two columns show how many bytes of data have been queued for the connection either from the other host or to it. The next column contains the hostname of the local host, followed by the port number of that connection. If the port is assigned to a known protocol, the protocol is displayed. The next column contains the name of the host on the far end of the connection, followed by port number or protocol if the port is known. The final column describes the state of the connection; ESTABLISHED means the TCP has been set up and is ready to accept data across it. In this example, we see the state SYN_SENT. This means that the host is trying to set up a connection to the far host; it has sent a SYN packet to set up the connection, but it has not yet received an acknowledgement. A complete table of the possible values of the connection state and their meaning is in Table 5-1.

Table 5-1: Netstat TCP connection states and their relevance to packet filters

Connection state	Meaning
SYN_SENT	System that initiates a TCP connection has sent SYN packet but has not received an acknowledgment. If a connection persists in this state, it means that either the remote host is down or that a packet filter is blocking the path to the remote end.
SYN_RECV	System has received a SYN packet for TCP setup and has sent an acknowledgment, but has not received a confirmation from the initiating system. If a connection persists in this state, it means that the remote system initiating the connection has gone down, or a packet filter is blocking return packets.
ESTABLISHED	TCP connection has been established. A packet filter is not blocking this connection.
FIN_WAIT1	Socket is closed, and connection is shutting down. Since this is part of the connection shutdown procedure, this means that packet filter did not block this connection.
FIN_WAIT2	Connection is shut down, and socket is waiting for shutdown from remote end. Since this is part of the connection shutdown procedure, this means that packet filter did not block the setup of this connection.
LAST_ACK	The remote end shut down, the socket is closed, and the host is waiting for acknowledgement. Since this is part of the connection shutdown procedure, this means that packet filter did not block the setup of this connection.
CLOSE_WAIT	System has received a FIN packet to terminate the connection. Since this is part of the connection shutdown procedure, this means that packet filter did not block the setup of this connection.
TIME_WAIT	TCP connection has closed, but the system waits for an interval before releasing the local port used. Since this is part of the connection shutdown procedure, this means that packet filter did not block the setup of this connection.

We wanted to be sure of SMTP's port behavior. The netstat output shows that SMTP goes from a high port (greater than 1023) on the system initiating the SMTP connection to a low port (less than or equal to 1023) on the SMTP port of the system it connects to. If a host must send and receive mail, you must permit access to the SMTP port going out and let systems connect to its SMTP port.

It is not always possible to get access to every host when debugging extended access lists. But if you can, running netstat and using the connection state information can be very helpful in tracking why a TCP-based application may not be running. On the application server, if you see that SYN packets have arrived and are acknowledged (the SYN_RECEIVED state), but no connections are in the ESTABLISHED state, then the application is not working because response packets cannot return to the client. From the client host, if you see that the connections to the application server are always in SYN_SENT mode, that means that either return packets are not coming back or SYN packets are not getting to the server.

Let's look at an example of how netstat can be used in this fashion. Recall from the last example that bad access lists prevented FTP services; netstat could have been used to find the problem, yielding the following output:

Active Internet connections
Proto Recv-Q Send-Q  Local Address         Foreign Address        (state)
tcp        0      0  server.ftp            client.1042           ESTABLISHED
tcp        0      0  server.ftpdata        client.1043           SYN_SENT

The FTP control connection is established, but the attempted TCP connection from the server to the client is not succeeding because it is stuck in the SYN_SENT state. This points toward the problem with setting up the data connection we identified.

Access list entry accounting

WARNING:   Access list entry accounting doesn't work with certain switching modes. Test this feature with your version of the IOS and the switch mode you want to use before you decide to rely on it for debugging.

There are a number of facilities on Cisco routers that allow you to gather more information to find where you may have allowed too much or too little into your packet filtering policy set. The first is one I discussed when talking about verifying correctness: access list entry accounting. Since Cisco routers count the number of times that extended access lists entries are used, you can use this feature to see why an access list may not be working. Consider a policy that says:

Disallow all FTP, Telnet, and SSH access to a host but allow SMTP access

The SSH protocol typically uses port 22, which means the forbidden ports are adjacent. Let's say we yielded to temptation and implemented an outgoing access list to the host with the following:

access-list 101 deny tcp any host 192.168.35.1 range 20 25
access-list 101 permit tcp any host 192.168.35.1 eq SMTP
interface ethernet 0
 access-group 101 out

However, after access list 101 is implemented, we find that SMTP to the host does not work. Access list accounting can help us find the problem. The command show access-list 101 yields something like this:

access-list 101
       deny tcp any host 192.168.35.1 range 20 25 (100 matches)
       permit tcp any host 192.168.35.1 eq smtp

The entry permitting SMTP has no matches, but the entry denying the range has many matches. This indicates that the first entry is matching all of the SMTP packets, so we know our problem is that the range of TCP ports is too large on our first entry. Access list 1 should be:

access-list 101 deny tcp any host 192.168.35.1 range 20 23
access-list 101 permit tcp any host 192.168.35.1 eq smtp

Access list accounting can show which entries are executed and which are not, indicating where an overly permissive or restrictive entry may lie.

TIP:   Sometimes when using access list accounting, you may want to reset all of the matches to 0. This can be useful, particularly when the counts become very high and it gets hard to remember if the counts changed. The command clear access list counters resets all of the match counts on all access lists to 0. When followed by an access list number or name, it clears only that particular access list.

IP accounting

Another facility useful for debugging is IP accounting. Looking at the access violation database can help you figure out exactly what is getting rejected and where in an access list this might be happening. It is particularly useful when there is a problem with IP address masking. In our FTP server example, let's add Telnet access to the FTP server from hosts 192.168.30.4 and 192.168.30.5. Let's say we implemented access lists as follows:

access-list 101 permit tcp any host 192.168.35.1 range 20 21
access-list 101 permit tcp host 192.168.30.5 host 192.168.35.1 eq telnet
access-list 102 permit tcp host 192.168.35.1 any established
access-list 102 permit tcp host 192.168.35.1 eq 20 any gt 1023
!
interface Ethernet0
 access-group 101 out
 access-group 102 in

Telnet to the FTP server doesn't work from 192.168.30.4 but does from 192.168.30.5. To debug the problem, we can turn on IP accounting for access list violations:

interface Ethernet 0
ip accounting access-violation

After trying to telnet, the output of show ip accounting access-violation is:

Source            Destination              Packets               Bytes ACL
192.168.30.4      192.168.35.1                   5               4343  101

This output shows that something is blocking 192.168.30.4 but not 192.168.30.5. A look at access list 101 shows that 192.168.30.4 is not in any entry or included in any mask, so there must be a problem in the entry allowing Telnet. Sure enough, the entry does not include 192.168.30.4. The correct access lists are:

access-list 101 permit tcp any host 192.168.35.1 range 20 21
access-list 101 permit tcp 192.168.30.4 0.0.0.1 host 192.168.35.1 eq telnet
access-list 102 permit tcp host 192.168.35.1 any established
access-list 102 permit tcp host 192.168.35.1 eq 20 any gt 1023

This example shows us how to find why packets that should be permitted are denied. The access violation database is also helpful in the opposite case, finding out why something is being permitted instead of denied. Let's say we implemented the previous Telnet policy with the following:

access-list 101 permit tcp any host 192.168.35.1 range 20 21
access-list 101 permit tcp 192.168.30.4 0.0.0.3 host 192.168.35.1 eq telnet
access-list 102 permit tcp host 192.168.35.1 any established
access-list 102 permit tcp host 192.168.35.1 eq 20 any gt 1023

To turn on transit packet accounting, we use the ip accounting command:

interface Ethernet0
 ip accounting output-packets
interface Ethernet1
 ip accounting output-packets

IP accounting is turned on for both interfaces so we can capture the traffic to and from the FTP server. The output of show ip accounting might look like this:

 Source                     Destination               Packets               Bytes
192.168.30.4           192.168.35.1                    9                     9052
192.168.30.5           192.168.35.1                    8                     8304
192.168.30.7           192.168.35.1                    6                     6572
192.168.35.1           192.168.30.4                   10                    12200
192.168.35.1           192.168.30.5                    9                    11206
192.168.35.1           192.168.30.7                    8                    10208

In addition to 192.168.30.4 and 192.168.30.5, host 192.168.30.7 has access. Since it is not much farther from the hosts we intended, the implication is that there is a bad mask. Indeed, that is the problem: the mask for access list 101's Telnet entry is too inclusive.

WARNING:   Like access list entry accounting, IP accounting doesn't work with certain switching modes. With some versions of the IOS on certain hardware platforms, turning on IP accounting can even disable packet forwarding! Test before using this feature.

Access list entry logging

The last debugging technique we'll discuss uses the router's logging capability. An extended access entry that ends in the keyword log sends information about the packet that matched the entry to the router's log. Recall the example in which we denied a number of TCP-based services to a host but allowed SMTP to it. Our original configuration had a problem with the first entry. To find the problem, we could enable logging with the following configuration:

access-list 101 deny tcp any host 192.168.35.1 range 20 25 log
access-list 101 permit tcp any host 192.168.35.1 eq SMTP
logging buffered
interface ethernet 0
access-group 101 out

The first line of access list 101 contains the keyword log at the end. This means that the information on each packet that matches this line is sent to the router's log. The configuration entry logging buffered instructs the router to send the output to the router's logging buffer. An alternative here is to substitute buffered with the IP address of a host that will receive the logging output through the syslog protocol. For example, the global configuration command logging 192.168.33.2 sends logging information to host 192.168.33.2 via syslog. The host receives syslog information at the debug information level.

When SMTP traffic goes through the router, it matches the first line of access list 101, and detailed information about the packets is sent to the router's log. The command show logging prints the router's log, in this case producing the following:

%SEC-6-IPACCESSLOGP: list 101 denied tcp 172.28.178.207 (1129) -> 192.168.35.1(25), 1 packet
%SEC-6-IPACCESSLOGP: list 101 denied tcp 172.28.178.207 (1130) -> 192.168.35.1(25), 1 packets

Logging output shows the source IP address, source port, destination IP address, and destination port for each packet. In this case, the output shows that SMTP traffic is getting blocked and immediately lets you know where the problem entry is.

You can systematically move the location of the logging entry from the start of an access list to the end to find where a given packet matches. If you are looking to see why packets are denied and no entry in a list generates a log output, then you are not permitting anywhere the packets you are testing (hitting the implicit deny).

A few notes about using the logging facility are important. If you use the router's logging buffer to store log output, it can fill up. You can clear it by issuing the command no logging buffered to turn logging off and then on again. Also, logging many events puts a load on the router that you need to monitor. If you are logging via syslog to a remote host, realize that the syslog input also puts a load on the host. If many logged events are coming in very quickly, there is the potential for the remote host doing syslog to lose events. Still, when used carefully, the logging facility can be a powerful tool for finding problems with extended access lists.

Route-filtering access control lists

As in previous sections, I start with a discussion of verifying route-filtering access control lists for correctness and then talk about debugging.

Checking for correctness

When checking the correctness of route filtering access control lists, you cannot rely on the technique we used previously (making sure that applications run correctly). Applications can run correctly even when taking a route that does not match the policy you are trying to implement. To make sure that route filtering access lists are correct, you need to use various diagnostic tools implemented in routers and hosts. The first tool is an examination of the routing table. The command show ip route displays a Cisco router's routing table. For routing policies that affect incoming routing updates, the show ip route command can verify that your routing policy implementation is correct.

Let's look at one of our previous routing policy implementations to see how we can use show ip route to verify an implementation's correctness. The first example in Chapter 4 deals with the network shown in Figure 5-3.

Figure 5-3. Restricting routes sent and received

Router 3, located in Site B, seeks to restrict the routes it receives from Site D. Since Site D uses only network 19.0.0.0/8 and does not transit any traffic from any other sites, Site B should hear about network 19.0.0.0/8 only in the routing updates it gets from Router 1. We implement the policy with the following:

access-list 1 permit 19.0.0.0
router eigrp 1000
network 192.168.3.0
distribute-list 1 in serial 1

How can we use show ip route to verify that this is correct? We have to make sure that network 19.0.0.0/8 is the only network learned from Router 1. Here is the relevant output from executing show ip route on Router 3:

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, * - candidate default
       U - per-user static route, o - ODR
 
Gateway of last resort is 172.28.1.5 to network 172.28.0.0
D EX 19.0.0.0/8 [170/2202624] via 192.168.3.2, 3w5d, Serial1
D EX 10.0.0.0/8 [170/2239232] via 172.28.1.5, 3w5d, FastEthernet0/0
D EX 192.168.4.0/24 [100/2174464] via 172.28.1.5, 3w5d, FastEthernet0/0
D EX 198.175.107.0/24 [100/2239232] via 143.183.152.251, 3w5d, FastEthernet1/0
C    192.168.1.0/24 is directly connected, Serial1
172.28.0.0/16 is variably subnetted, 226 subnets, 4 masks
C       172.28.1.0/26 is directly connected, FastEthernet1/0

The first part of this output explains the different codes used in the routing table output. For example, the code in front of a route describes how the route was learned. The first line of the entry with network 19.0.0.0/8 begins with a D, meaning that a route has been learned via EIGRP. Table 5-2 contains a summary of possible codes and what they mean. The next part of the output describes default routing--where the router sends packets to networks not in its routing table. I'll describe this in detail later.

Table 5-2: Routing protocols and their meanings

Route code	Meaning
C	Network is directly connected to the router
S	Route was statically configured
I	Route was learned via IGRP
R	Route was learned via RIP
M	Route was learned by the mobile IP protocol
B	Route was learned by BGP
D	Route was learned by EIGRP
EX	Route is an EIGRP external route
O	Route was learned by OSPF
N1	Route is an OSPF NSSA external route type 1
N2	Route is an OSPF NSSA external route type 2
E1	Route is an OSPF external route type 1
E2	Route is an OSPF external route type 2
E	Route was learned by EGP
i	IS-IS
L1	IS-IS level-1
L2	IS-IS level-2
*	Route is a candidate default route
U	Route is a per-user static route
o	Route is an on demand route

The last part is a series of lines that list the contents of the routing table. Each line describes the path to each distinct network in the routing table, and each is composed of a number of distinct parts. The routing protocol used to learn the path to that network appears before the network number. The next part of each line, within the brackets, is the administrative distance and the routing protocol metric. In this example, you can see that the networks 19.0.0.0/8 and 10.0.0.0/8 have an external EIGRP administrative distance of 170. If a network is directly connected to the router, then the administrative distance/metrics section is omitted (connected networks have an administrative distance of 0). Next is the next hop, where the router forwards packets bound for that network. You can see that packets bound for network 19.0.0.0/8 should be forwarded to 192.168.3.2. Packets for network 10.0.0.0/8 should be forwarded to the router with IP address 172.28.1.5. Directly connected networks list the interface that the networks are connected to instead of an IP address. After the next hop is a string indicating how long the router has known the route. Routing protocols such as IGRP and RIP broadcast their routes only periodically, so this number is usually pretty low. EIGRP and BGP, routing protocols that send out routing updates only when network topologies change, have large values here, possibly days or weeks. Static routes and connected routes do not list the time since these routes are not learned dynamically. The last part of the routing table entry description is additional route information. Some routing protocols list what router interface the packets for the network will take. For example, packets bound for network 19.0.0.0/8 will travel out through interface serial 1.

How we can use this input to check on our policy implementation? First, let's look at the routing information for network 19.0.0.0/8. Our policy requires that this is the only route learned from Site D. In looking at the show ip route output, note that the only route learned via EIGRP from Site D's Router 1 is 19.0.0.0/8, and the next hop for the network is 192.168.3.2, the serial interface of Router 1. It seems that our implementation is correct. Is it really? If Router 1 advertised only the route 19.0.0.0/8, the output would be the same whether or not access list 1 was defined and applied with a distribute-list command. To be more certain, you need to make sure that the access lists are really used, with the show ip interface command or by checking the router's configuration.

Limiting routing output

The command show ip route can produce a lot of output, depending on the size of the router's routing table. There are a number of command qualifiers that can limit the output to what a network administrator finds useful. For instance, show ip route followed by a network number provides detailed routing information for that network. For example, typing:

show ip route 19.0.0.0

produces output like this:

Routing entry for 19.0.0.0/8
  Known via "eigrp 1000", distance 170, metric 2202624, type external
  Redistributing via eigrp 1000
  Last update from 192.168.3.2 on Serial0, 3w5d ago
  Routing Descriptor Blocks:
  * 192.168.3.2, from 192.168.3.2, 3w5d ago, via Serial1
      Route metric is 2202624, traffic share count is 1
      Total delay is 22470 microseconds, minimum bandwidth is 1544 Kbit
      Reliability 1/255, minimum MTU 1500 bytes
      Loading 255/255, Hops 6

As you can see, detailed routing information about the network 19.0.0.0 route table entry is displayed. Information about how the route is redistributed and the components that make up the route metrics (such as delay, bandwidth, reliability, and loading) is also included.

Filtering router output Commands such as show access-list or show ip route can generate so much output that they can be difficult to use for debugging. Versions of the Cisco IOS starting 12.0(1) T allow an administrator to include or exclude the output containing a specific string. The output modifier "|", when followed by the keywords inc, exc, or begin, plus a string, can be used to modify the output. For example, the command show ip route | inc 172.28 displays all the lines of show ip route that contain the string 172.28. If we use the command show ip route | exc 172.28, the lines that do not have 172.28 are be shown, while show ip route | inc begin 172.28 shows the lines that begin with 172.28. show ip access-list | inc match shows all of the access list entries that have a match. Other uses for these output modifiers include displaying the routes to all the subnets of a particular network, showing all of the access list entries that affect traffic to a specific destination, and displaying data in the IP accounting database that pertains to a specific IP address.

We can also display only the routes learned by a specific routing protocol. For example, the following command:

show ip route eigrp

shows all of the routes learned by EIGRP:

D EX 19.0.0.0/8 [170/2202624] via 192.168.3.2, 3w5d, Serial1
D EX 10.0.0.0/8 [170/2239232] via 172.28.1.5, 3w5d, FastEthernet0/0
D EX 192.168.4.0/24 [100/2174464] via 172.28.1.5, 3w5d, FastEthernet0/0

Typing the following:

show ip route connected

produces:

C    192.168.1.0/24 is directly connected, Serial0
172.28.0.0/16 is variably subnetted, 226 subnets, 4 masks
C       172.28.1.0/26 is directly connected, FastEthernet0/0

Default gateway information and an explanation of routing codes are not displayed when the routing protocol is added.

In Chapter 4, I showed how you could use the default route to limit the number of routes that need to be accepted. In our example (see Figure 4-12), we limited incoming routes to default network 172.28.0.0 with an access list and distribute-list on Router 1:

default-network 172.28.0.0
access-list 1 permit 172.28.0.0
router eigrp 1000
network 192.168.3.0
distribute-list 1 in serial 0

In this case, the default network information is very useful for verifying the correctness of the policy implementation. A correct implementation of the policy yields show ip route output such as the following:

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, * - candidate default
       U - per-user static route, o - ODR
 
Gateway of last resort is 192.168.3.3 to network 172.28.0.0
D*EX 172.28.0.0/16 [170/2202624] via 192.168.3.3, 3w5d, Serial0
C    192.168.3.0/24 is directly connected, Serial0
19.0.0.0/8 is variably subnetted, 200 subnets, 2 masks
C       19.1.1.0/24 is directly connected, FastEthernet0/0

The default network 172.28.0.0/16 has been learned from Router 3's 192.168.3.3 interface. We can see that it is properly handled because it's the only network learned from Router 3. It is listed as the default network in the default routing information section, and we can also see that it is the default network because its routing table entry has been flagged with an asterisk. With this information, we can conclude that the policy implementation is correct.

Verifying the correctness of access lists in outbound distribute-list statements

So far, we have only looked at verifying the correctness of access lists referenced by inbound distribute-list statements. How would you check for the correctness of access lists used in outbound distribute-list statements? One way is to look at the routing table of a router receiving the filtered routes. In Chapter 4, I showed an example of Site D filtering the routes it distributes in order to prevent any routing problems within Site D from spreading to other sites. Only network 19.0.0.0/8 should be advertised from Site D, so I configured the following on Router 1:

access-list 2 permit 19.0.0.0
router eigrp 1000
network 192.168.3.0
distribute-list 2 out serial 0

How would you verify that this configuration implements our policy? Looking at Router 1's routing table does no good since it doesn't listen to its own updates. The routing table we need to look at is on Router 3. If we disable the inbound distribute-list on serial 1 of Router 3, we can look at its routing table to see if 19.0.0.0 is the only route sent from Router 1. For verifying policies using outbound distribute-lists, examining the routing tables of the routers receiving the route updates is a good technique. Another method is to use the debug facility and watch routing updates sent from the router to make sure the proper routes are being advertised. I'll talk about using debug later in this chapter.

Verifying that hosts receive correct routing information

This technique also applies to hosts receiving routing updates. If you are filtering routes sent to hosts, checking the hosts routing table enables you to see if your policy implementation is correct. To look at a host's routing table, do the following command:

netstat -rn

To demonstrate this command, let's say we advertise routes via RIP to the hosts on segment 172.28.1.0/24.

For a host on that segment, netstat would produce output like the following:

Routing tables
 
Internet:
Destination         Gateway            Flags    Refs      Use  Interface
default             172.28.1.5         UGS         0  1519419  de0
10                  172.28.1.5         UG          0      429  de0
192.168.3.0         172.28.1.5         UG          0        5  de0
192.168.4.0         172.28.1.6         UG          0       12  de0
19                  172.28.1.6         UG          0   123543  de0
172.28.1.0          172.28.1.10        UG          0     2386  de0

The first column lists networks or hosts, and the second column shows the next hop or gateway to reach the host or network from the first column. Thus the route to network 10.0.0.0/8 goes through the router at 172.28.1.5, and the route for 19.0.0.0/8 goes out through the router at 172.28.1.6. Traffic to subnet 172.28.1.0 is local, so the gateway is listed as the host's own IP address, 172.28.1.10. The third column contains various flags associated with the route. This column is important. When only the UG flag is listed, it means that the host has learned routes from routing protocols. The presence of a capital "S" means that the route has been statically assigned. Like static routes on routers, static routes on hosts are not learned dynamically.

Traceroute

The technique of looking at adjacent routing tables has limitations. Although a particular route for a network may not be present in a routing table, that does not necessarily mean it is not being advertised. A route with a better metric may be advertised from elsewhere, and only when that second route goes away does the first route appear. Applying this potential trap to our situation, let's say that for some reason, Router 1 also advertises network 10.0.0.0/8, and Router 3 hears this advertisement. If Router 3 hears an advertisement with a better metric for network 10.0.0.0/8, the route to 10.0.0.0/8 through Router 1 does not appear in the routing table. Only when the second advertisement goes away for some reason is the bad advertisement noticed. In general, the technique of looking at routing tables is a necessary but not sufficient condition for correctness.

Another limitation of this technique is that we've been assuming we have access to neighboring routers. In situations where routers under your control border routers under someone else's administrative control, this can often be a problem since the other network administrators may be unwilling or unable to grant you SNMP read access, Telnet access, or the passwords to these routers. In that case, you have to resort to other tools like traceroute, which shows the path of a packet to its destination. Traceroutes are initiated with the trace command, followed by the hostname or IP address of the host or router to which you want the path to go. Recall that the example network we've been working with is part of a larger network, shown in Figure 5-4.

Figure 5-4. A corporate network

Let's say that an interface of Router 6 has an IP address of 10.1.1.2 and that the Router 1 interface going to Site B has an IP address of 192.168.3.2. If we wanted to traceroute from Router 1 to Router 6, we issue the following command:

trace 10.1.1.2

which produces the following output:

Type escape sequence to abort.
Tracing the route to router6-e0.somecompany.com (10.1.1.2)
 
  1 router3-s1.somecompany.com (192.168.3.3) 4 msec 3 msec 2 msec
  2 router4-e0.somecompany.com (172.28.1.6) 4 msec 6 msec 5 msec
  3 router6-s0.somecompany.com (192.168.4.2) 20 msec * 16 msec

The first hop from Router 1 is to Router 3, at IP address 192.168.3.3. The next hop goes to an interface of Router 4, at IP address 172.28.1.6. The final hop is to the serial 0 interface of destination Router 6, at 192.168.4.2. The three items listed after the IP address of each hop describe the time taken for each traceroute probe packet to get a response. If no response is reached, an asterisk is printed.

Running traceroute on hosts Most Unix and Windows systems either have the traceroute command or can run it. From Unix systems with traceroute, use the traceroute command followed by the host name or IP address. On Windows 95, 98, or NT, use the command tracert followed by the hostname or IP address. If you tracert from host 172.28.1.11 to 10.1.1.2, you should see output like the following: Tracing route to router6-e0.somecompany.com [10.1.1.2] over a maximum of 30 hops:   1 1 ms 0 ms 0 ms router4-e0.somecompany.com [172.28.1.6] 2 18 ms 19 ms 17 ms router6-s0.somecompany.com [172.28.1.6] This version of traceroute differs in that the response times are listed before the hostname. This version also uses ICMP from probe packets instead of UDP.

Traceroute is useful for verifying a number of routing policy implementations. For outbound distribute-list and outbound offset-list statements, doing a traceroute from a remote network to an IP address in a network you are advertising can show you if the policy you are implementing for the network is working properly. You can see whether the packets to a network are taking the path your policy dictates. If you don't have access to a host or a router in a remote network to do a traceroute back to the network, you can have someone in another organization do the traceroute for you. On the Internet, there are a number of publicly available traceroute servers that allow you to traceroute back to your own network or to other networks. For an inbound distribute-list or offset-list, doing a traceroute to a network you have a policy about can show you whether the next hop is appropriate.

Let's see how we can use traceroute to verify the correctness of some of our policies. Recall that we defined a policy that limits networks advertised out of Site D (on Router 1) to network 19.0.0.0/8 and a policy that limits routes received from Site D (on Router 3). The network administrators of Site D could ask someone at another site, such as Site B or Site E, to do a traceroute to a host in network 19.0.0.0/8. If the traceroute goes through Router 1, then Site D knows that Router 1's route filters have not accidentally filtered out its network 19.0.0.0/8 advertisements and that Router 3 has also properly filtered incoming advertisements.

When you execute the command trace with no arguments from enable mode, you have access to a number of useful options. In particular, you can choose the source IP address of the traceroute probes. Since traceroute responses are sent back to the source address (see the upcoming sidebar "Traceroute: How it functions and how to filter it"), you can use this option to determine if a destination has a route back to the network the source address belongs to. The source address must be an interface of the router sending the traceroute. We can use this to see whether Site B can reach Site E's network 10.0.0.0/8 and whether Site E has a route back to Site B's network 19.0.0.0/8. We should use traceroute from Router 1 as follows:

Traceroute: How it functions and how to filter it Traceroute is a very useful tool for debugging route filtering. It is easy to stop it from functioning with packet filters, however. Traceroute works by sending out a number of UDP or ICMP packets (the default is typically 3, although this can be set) with increasing Time To Live (TTL) parameters. To learn the first hop, the host or router doing traceroute sends out a packet with a TTL of 1. When this packet arrives at the first router hop, the TTL has expired, so the first hop router sends back an ICMP TTL Exceeded packet back to the sender. Thus the sender learns the identity of the first hop. The sender then sends out a packet with a TTL of 2. The packet expires at the second hop, so the second hop router sends an ICMP TTL Exceeded packet back to the sender. This process continues until the destination is reached. From this description, you can see that the sender needs to send out UDP or ICMP (depending on the implementation) in order for traceroute to function. If the sender uses UDP, those ports must be permitted. Usually, the destination port used can be set, and the source port is usually above 1023. The sender must also be able to receive ICMP time-exceeded packets. In addition, since the packets with increasing TTL have a source address of the interface of the router or system used, be careful that the IP address of the particular interface used to send packets has permission to send out and receive those packets. For example, if we say that access list 101 is assigned to filter incoming packets to a router interface leading to a host at 192.168.35.1, that host will do traceroute, and that access list 102 is assigned to filter outgoing packets, then the following extended access list entries will allow traceroute from any host: access-list 101 permit udp host 192.168.35.1 any access-list 102 permit icmp any host 192.168.35.1 ttl-exceeded This assumes we use UDP for the increasing TTL packets. Letting out all UDP is very broad, and you may want to narrow the number of UDP ports allowed out. The default UDP port that traceroute starts with is 33434. Some traceroute implementations increase the destination port number with each set of probes, so the following two access list entries should work: access-list 101 permit udp host 192.168.35.1 ge 33434 access-list 102 permit icmp any host 192.168.35.1 ttl-exceeded

Router1# trace
Protocol [ip]:
Target IP address: 10.1.1.2
Source address: 19.1.1.1
Numeric display [n]:
Timeout in seconds [3]:
Probe count [3]:
Minimum Time to Live [1]:
Maximum Time to Live [30]:
Port Number [33434]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Type escape sequence to abort.

You can see the many options possible with traceroute. We can set the port number used, probe count, and a number of different options. In this example, we only use the option for setting a source address. We traceroute from interface IP address 19.1.1.1 on Site B to address 10.1.1.2 on Site E. If we see output like the following:

Tracing the route to router6-e0.somecompany.com (10.1.1.2)
 
  1 router3-s1.somecompany.com (192.168.3.3) 4 msec 3 msec 2 msec
  2 router4-e0.somecompany.com (172.28.1.6) 4 msec 6 msec 5 msec
  3 router6-s0.somecompany.com (192.168.4.2) 20 msec * 16 msec

we know that Site B can reach Site E and vice versa.

Debugging routing policies with access list accounting

One of the more difficult routing policy implementations to verify is route preference. Let's look at a policy we implemented in Chapter 4. As a reminder, Figure 5-5 shows a network where we prefer that traffic use one path and then another if the first is down.

Figure 5-5. A network with a preferred routing scheme

This network uses RIP as its routing protocol. The preferred path between network 10.0.0.0/8 is through Router 1, but if that path is down, traffic should go the 56-Kb path. In this example, the only router that we control is Router 2. In Chapter 4, I implemented the policy as follows:

access-list 1 permit 10.0.0.0
access-list 2 permit 172.20.0.0
!
router rip
 network 192.168.1.0
 network 192.168.2.0
 network 172.20.0.0
 offset-list 1 in 3 serial 0
 offset-list 2 out 3 serial 0

How can we test this implementation of routing preferences? If we can find a time when network downtime is tolerated, one way is to use the methods described previously to verify that there is the proper connectivity between the two networks. Once the preferred path is known to work, you can break the connectivity on the path through Router 1 and test if traffic uses the next preferred path.

Unfortunately for most network administrators, the opportunity to have some network downtime for testing is not readily available. In this example, we have access only over Router 2. How would we verify that we have implemented the proper policy? Fortunately for us, there are a number of facilities on Cisco routers that can help. Since we have access only to Router 2, we need to verify that our changes to routing advertisements are being properly acted upon. Traffic between networks 10.0.0.0/8 and 172.20.0.0/16 should be coming in and out of serial 1. Looking at our routing table verifies that traffic from network 172.20.0.0/16 to network 10.0.0.0/8 is going the right way. We need to verify that traffic goes from network 10.0.0.0/8 to network 172.20.0.0/16.

One way we can do this is with an incoming extended access list combined with access list accounting. We use an access list to monitor if traffic is coming into the interface with the following configuration commands:

access-list 101 permit ip 10.0.0.0 0.0.0.255 172.20.0.0 0.0.255.255
access-list 101 permit ip any any
!
int serial 1
 access-group 101 in

If traffic is coming into serial interface 1 from network 10.0.0.0/8 bound for network 172.20.0.0/16, the fact that the traffic has gone through is recorded by access list accounting. The command show ip access-list 101 yields:

access-list 101
     permit ip 10.0.0.0 0.255.255.255 172.20.0.0 0.0.255.255 (100 matches)
     permit ip any any (10 matches)

You can see that traffic is coming in the way we intended. The access list does not interfere with traffic moving through the interface, as it lets all packets through but counts traffic going by. You could put a similar access list on serial interface 0 to verify that traffic from network 10.0.0.0/8 to network 172.20.0.0/8 is not going in through that interface.

Verifying routing protocol activity using debug

The previous access list technique lets us know if incoming traffic is okay, as it goes into the interface preferred by the policy. What about our implementation of backup routing? We need to make sure that our route advertisements are okay. To do that, we use the debug feature of the Cisco router, which allows us to see key events and data. To see debug output, we first need to issue the command terminal monitor if we are not using the router's system console port. This command sends debug output to our terminal session. Debug output is also sent to the router's console by default.

To see the debug output, we have to turn on routing debug output. Since we are using RIP as a routing protocol, we use the command debug ip rip, which limits debugging output to only that concerning RIP. When we issue this command, we start to see output like this:

RIP: received update from 192.168.2.1 on Serial0
     network 10.0.0.0 in 1 hops
     network 192.168.3.0 in 1 hops
RIP: received update from 192.168.1.1 on Serial1
     network 10.0.0.0 in 2 hops
     network 192.168.3.0 in 1 hops
RIP: sending update to 192.168.2.255 via Serial0 (192.168.2.2)
     network 172.20.0.0, metric 4
     network 192.168.1.0, metric 1
     network 192.168.3.0 metric 2
RIP: sending update to 192.168.1.255 via Serial1 (192.168.1.2)
     subnet 172.20.0.0, metric 1
     network 192.168.2.0, metric 2

The debug output shows that Router 2 is sending the correct output out through serial interface 0. Network 172.20.0.0/16 is advertised with a metric of 4, demonstrating that the offset of 3 is being added to the route metric as intended. Advertisements of network 10.0.0.0/8 are coming in with a metric of 1 through serial interface 0 and with a metric of 2 through serial interface 1. Since you see the correct advertisements coming in, you are reasonably sure that your router can use the path out through serial 0 as a backup. The offset-lists take care of increasing the metric to make the path through Router 1 the preferred route. This can be verified by looking at the route table and seeing the route to network 10.0.0.0/8.

To turn off debugging, use the undebug command. In this case, you invoke it with undebug ip rip. You could also use undebug all to turn off all debugging. If you don't want to receive debug output while using a terminal, use the command terminal no monitor. It is a good idea to make sure that all debugging output is off when you invoke the terminal no monitor command.

There are two important caveats of the debug command. First, don't overdo it with debug information. Too much information can be impossible to process meaningfully. A router with multiple interfaces sending and receiving periodic routing updates can scroll information so fast that you cannot see it unless you are recording your terminal session. Second, debug output is process-switched, so it places a burden on the router's main processor. Too much debug output can cripple a router and render it unusable.

Viewing routing topology

Using debug to see metric values works well for routing protocols such as RIP and IGRP that send out periodic updates of all of its routing information, but how would you verify routing preferences for a routing protocol such as EIGRP that only sends routing updates when conditions change? Recall that the routing table only contains the most preferred route, not any other routing information. Fortunately, these protocols typically have commands associated with them that display the topology learned by the routing protocol. For EIGRP, the command is show ip eigrp topology, so if you use EIGRP instead of RIP in the previous example, the show ip eigrp topology command produces output like this:

P 10.0.0.0 255.0.0.0, 2 successors, FD is 2236672
         via 192.168.2.1 (2236672/2234624), Serial1
         via 192.168.1.1 (2237184/2234624), Serial0
P 192.168.3.0 255.255.255.0, 2 successors, FD is 2236672
         via 192.168.2.1 (2236672/2234624), Serial1
         via 192.168.1.1 (2237184/2234624), Serial0

Unlike with the show ip route command, you can see the different paths to each network and the metrics (the first number within the parentheses) for each path. The path through Router 1 (via 192.168.2.1) is preferred, since it has a lower metric of 2236672. You can also see the backup paths over the 56-Kb link have a higher metric of 2237184. In this way, you verify that the preferred and the backup path advertisements are coming as intended. show ip eigrp topology has the limitation of being able to look only at incoming route updates, but it is still useful, as it gives you visibility into the route preferences. Other routing protocols such as OSPF have similar commands.

Debugging route-filtering access lists

Like router resource access lists and extended access lists, a route-filtering access list can go wrong in two ways: the access list denies a route that should be permitted, or it permits a route that should be denied. Just as router resource access lists and extended access lists can seem to fail because of routing problems, route filtering access lists can seem to fail because of other issues, such as packet filtering extended access lists. In this section, I discuss how to find problems with router filtering access lists. I also talk extensively about how factors other than actual route-filtering access list errors can make it look like there are problems.

In this section, I focus on the more complex debugging case in which a route you want is missing in a routing table. When a route is not present, there may be either no path or the wrong path to a given network. Let's look at the example shown in Figure 5-5. What if the routing table of Router 2 looked like the following?

C 172.20.0.0/16 is directly connected, Ethernet0

Network 10.0.0.0/8 is not in the routing table at all, so clearly something went wrong. What could have happened? Before concluding there is an access list error, you need to rule out several other possibilities. The connectivity to network 10.0.0.0/8 could be down. If the serial lines connecting to interfaces serial 0 and serial 1 are down, no routing information can be learned through them. In this case, the show interface command would show the following for these two interfaces:

Serial0 is down, line protocol is down
  Hardware is HD64570
  Description: To network 10 via 56Kbit path
  Internet address is 192.168.1.2/24
  MTU 1500 bytes, BW 56 Kbit, DLY 20000 usec, rely 255/255, load 1/255
  Last input 03:25:35, output 03:25:35, output hang never
  Last clearing of "show interface" counters never
  Queueing strategy: fifo
  Output queue 0/80, 0 drops; input queue 0/100, 0 drops
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     3976767 packets input, 371394120 bytes, 0 no buffer
     Received 732003 broadcasts, 120 runts, 0 giants, 0 throttles
     1235 input errors, 2 CRC, 680 frame, 0 overrun, 0 ignored, 0 abort
     0 input packets with dribble condition detected
     3586981 packets output, 1378381771 bytes, 0 underruns
     2 output errors, 0 collisions, 615 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
Serial1 is administratively down, line protocol is down
  Hardware is HD64570
  Description: To network 10 via T1 path
  Internet address is 192.168.2.2/24
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec, rely 255/255, load 1/255
  Encapsulation HDLC, loopback not set, keepalive set (10 sec)
  Last input 03:26:05, output 03:26:05, output hang never
  Last clearing of "show interface" counters never
  Queueing strategy: fifo
  Output queue 0/80, 0 drops; input queue 0/100, 0 drops
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     89764824 packets input, 4874394145 bytes, 0 no buffer
     Received 752342 broadcasts, 75 runts, 0 giants, 0 throttles
     52 input errors, 1 CRC, 20 frame, 0 overrun, 0 ignored, 0 abort
     0 input packets with dribble condition detected
     53586321 packets output, 61278388290 bytes, 0 underruns
     6 output errors, 0 collisions, 2 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

The show interface command generates a lot of output, but I'll mention only key parts that are relevant to debugging route-filtering access list problems. The first line of output for each interface is usually the most important. It tells you the interface name, whether it is up or down, and whether the line connected to the interface is down. Often, if the line protocol is down, the interface will be listed as down as well. In the previous example, both interfaces are down. Serial interface 1 is said to be administratively down. This means that the interface was manually shut down. The line beginning with Description: displays whatever the person who configured the router decided to mention about the interface. It is often useful for determining the function of the interface and what it connects to.

Below the description is the interface's IP address and mask. Underneath is an important line that describes key properties of the line connected to the interface. These properties, such as bandwidth (BW) and delay, are used by some routing protocols as part of metric calculations. In the example, this line shows which interface has the 56-Kb line and which has the 1.544-Mb line. Two lines below that line are times listing the last input and output. This line is useful for listing when an interface or the line connected to it goes down.

The show interface command output reveals that what at first appeared to be a routing problem was actually caused by interfaces being down. Another clue that the interfaces are down is that the directly connected networks, 192.168.1.0/24 and 192.168.2.0/24, are not listed in the route table. If the interfaces were up, then the two networks would have been listed in the routing table.

When the wrong route is present

Another possibility is that the wrong route might be present. The show ip route command yields the following:

C 172.20.0.0/16 is directly connected, Ethernet0
C 192.168.1.0/24 is directly connected, Serial0
R 10.0.0.0/8 [120/1] via 192.168.1.1, 0:29, Serial0

There is a route to 10.0.0.0/8, but it uses the backup path through serial 0 instead of the path through serial 1. What could have happened? In this case, it is possible that the line connected to interface serial 1 is down, and show interface serial1 would show the following:

Serial1 is down, line protocol is down

Let's now say that serial interface 1 is down, so the route to network 10.0.0.0/8 uses serial interface 0. Network 192.168.2.0/24 is not in the route table. This is additional evidence of serial interface 1 being down.

What if 192.168.2.0/24 did appear in the route table yet the route was still wrong? We'd then see the following:

C 172.20.0.0/16 is directly connected, Ethernet0
C 192.168.1.0/24 is directly connected, Serial0
C 192.168.2.0/24 is directly connected, Serial1
R 10.0.0.0/8 [120/4] via 192.168.1.1, 0:29, Serial0

Here, both serial interfaces must be up, since the directly connected networks on both interfaces are listed in the route table. However, this does not necessarily mean there are problems with our access list. Notice that the metric for network 10.0.0.0/8 is 4, which is what it should be after we increase its value with the offset-list command. This implies that the routing information coming into the serial 0 interface is being handled correctly, so it could be that the router on the other side of the 1.544-Mb line is down. We can confirm this by doing a ping of router 1's 192.168.2.1 interface. If ping reveals that it is down, then this problem has prevented the preferred route from entering Router 2's routing table.

If ping reveals the interface at 192.168.2.1 is up, then there are a number of possibilities why the correct route is still not there. Router 1's connectivity to network 10 might be down. Also, Router 1's administrators may have accidentally turned off sending routes to Router 2, through the passive-interface statement. Strange as it may seem, that actually does happen occasionally and can be detected using the debug command. You would notice that Router 1 seems to send no routing updates even though it is up.

Stopping routing updates with extended access lists

Make sure that you don't cause problems yourself. One way that routing updates can be ignored is with an overzealous incoming packet filter. If Router 2 makes the following attempt at anti-spoofing:

access-list 101 deny ip 172.20.0.0 0.0.255.255 any
access-list 101 deny ip 192.168.2.0 0.0.0.255 any
access-list 101 permit ip any any
interface serial 1
access-group 101 in

then all incoming routing updates are suppressed. At other times you might forget static routes you have set previously. If show ip route produces the following:

C 172.20.0.0/16 is directly connected, Ethernet0
C 192.168.1.0/24 is directly connected, Serial0
C 192.168.2.0/24 is directly connected, Serial1
S 10.0.0.0/8 [1/1] via 192.168.1.1, Serial0

a static route excludes any routing advertisement of network 10.0.0.0 since static routes have a higher administrative distance then any dynamic routing protocol.

If all else fails You may encounter a situation where your access lists don't work even though there is no obvious reason. It may seem that your access lists are being totally ignored, or that your access list stops all traffic through an interface despite clearly permitting certain types of traffic. When you encounter this type of error, there may be a problem with your particular IOS. After you have exhausted all other debugging possibilities, check the Cisco web site for bugs in your IOS (http://www.cisco.com/support/bugtools). There are IOS versions in which access lists do not function correctly. The Cisco Bug Navigator? can point out the IOS versions that have problems and the ones that have fixed those problems. If you don't find your problem, open up a case with the Cisco Technical Assistance Center (TAC). They should be able to help. Note that both of these options require a support contract, which I strongly recommend if you depend on Cisco access lists for mission critical applications.

When access lists are used incorrectly

Sometimes, access lists are correct but not used correctly. If you implement the access lists like this:

access-list 1 permit 10.0.0.0
access-list 2 permit 172.20.0.0
router rip
network 192.168.1.0
network 192.168.2.0
offset-list 1 in 1 serial 0
offset-list 2 out 1 serial 0

show ip route reveals the following:

C 172.20.0.0/16 is directly connected, Ethernet0
C 192.168.1.0/24 is directly connected, Serial0
C 192.168.2.0/24 is directly connected, Serial1
R 192.168.3.0/24 [120/1] via 192.168.2.1, 00:23, Serial1
R 10.0.0.0/8 [120/2] via 192.168.1.1, 00:23, Serial0
  10.0.0.0/8 [120/2] via 192.168.2.1, 00:23, Serial1

Notice that there are two routes associated with network 10.0.0.0/8, one for each of two possible paths, and they both have the same route metric. This is an indication that the offset used in the offset-list line is not high enough. Since the metrics to both paths are the same, the router installs both routes.

When route-filtering access lists are wrong

I have shown how different problems can make you think there are access lists problems when there are not. Let's look at a situation where there actually are problems with the access lists themselves. If we implement our policy in the following way:

access-list 1 permit 19.0.0.0
access-list 2 permit 172.20.0.0
router rip
network 192.168.1.0
network 192.168.2.0
offset-list 1 in 3 serial 0
offset-list 2 out 3 serial 0

show ip route reveals the following:

C 172.20.0.0/16 is directly connected, Ethernet0
C 192.168.1.0/24 is directly connected, Serial0
C 192.168.2.0/24 is directly connected, Serial1
R 192.168.3.0/24 [120/1] via 192.168.2.1, 00:23, Serial1
R 10.0.0.0/8 [120/1] via 192.168.1.1, 00:23, Serial0

The path to network 10.0.0.0/8 goes through serial 0, yet its metric is only 1. It looks like access list 1 has not put network 10.0.0.0 into its policy set of routes that will have its route metrics increased. A look at access list 1 reveals that network 19.0.0.0/8 is in the only entry instead of network 10.0.0.0/8. Once corrected, the proper route should be installed in the routing table.

Back to: Cisco IOS Access Lists