Summarize Router Throughput in Packets/Second

Here's a quick one-liner to summarize the current packets-per-second throughput on an entire IOS router:

$ ssh 10.1.2.3 'sh int summ' | awk '/^*/{RXPPS+=$8;TXPPS+=$10} END {print "RXPPS=" RXPPS,"TXPPS=" TXPPS}'
jswan@10.1.2.3's password:
RXPPS=1613 TXPPS=1539

Zone Based Firewall Configuration Example

There aren’t a ton of examples available for IOS Zone-Based Firewall configurations, so I thought I’d put up one with which I’ve been working recently.

My test network looks like this:

R4 ------------------- R5 ----------------------- R6

R4 Loopback: 4.4.4.4

R4 to R5 link: 1.1.45.0/24

R5 Loopback: 5.5.5.5

R5 to R6 link: 1.1.56.0/24

R6 Loopback: 6.6.6.6

My goal is to express this policy:

1. Hosts in group X on the OUTSIDE network can initiate sessions to hosts on the INSIDE network. Return traffic for those sessions should be statefully permitted. In the lab, group X is represented by R4's loopback address.
2. Any host on the INSIDE network can initiate sessions to hosts in group X on the OUTSIDE network. Return traffic for those sessions should be statefully permitted. In the lab, the inside hosts are represented by R6's loopback address.
3.  Permitted traffic should be logged. All other traffic should be dropped and logged.

The network on which the production version of this lab test will be deployed currently uses extended access-lists to implement the policy. Access-lists have the advantage of being familiar to anyone with a basic knowledge of IOS, but they have a lot of disadvantages:

1. Hard to read and troubleshoot as they grow. 
2. No stateful awareness. Trying to retrofit statefulness onto extended ACLs to allow return traffic requires ugly hacks with source port restrictions, filtering on ACK bits, etc. This path eventually leads to mind-numbing troubleshooting problems, and is less than optimal in its security. 
3. Extended ACLs don’t have application-layer awareness.

There are several tools in IOS to facilitate stateful traffic inspection, but Zone-Based Firewalls are the newest and most flexible, so it made sense to use them.

Most of the examples I found via Google show the simple case of inside hosts having unfettered access to the outside, and outside hosts having no access to the inside--a classic stateful firewall design. My case is only slightly more complex, but it still took me a couple of tries to make it work.

First, I built access-lists defining the traffic to be allowed. I decided to use the relatively new IOS object-group support to make it easier to add and delete hosts on the respective networks:


object-group network OG_INSIDE_HOSTS
  host 6.6.6.6
object-group network OG_OUTSIDE_ALLOWED
  host 4.4.4.4
ip access-list extended INSIDE_TO_OUTSIDE
  permit ip object-group OG_INSIDE_HOSTS object-group OG_OUTSIDE_ALLOWED
ip access-list extended OUTSIDE_TO_INSIDE
  permit ip object-group OG_OUTSIDE_ALLOWED object-group OG_INSIDE_HOSTS


You might notice that there’s no “deny ip any any log” statement in these ACLs, which is strange given requirement #3 above. It turns out that when using ZBFW, you configure drop logging elsewhere; I’ll cover that below.

I couldn’t find a way to express protocol inspection policy and IPv4 address policy in the same class-map, so I had to use a hierarchical configuration. To express the protocol inspection policy, I built a class-map to define the layer 4 protocols that will be inspected by the firewall:


class-map type inspect match-any CM_INSPECTED_PROTOCOLS
  match protocol icmp
  match protocol tcp
  match protocol udp


This class-map does simple generic TCP/UDP/ICMP inspection, but it could easily be extended or rewritten to use much more complex inspection rules.

Next, I built class-maps for each direction that match the appropriate ACL and the inspection policy configured above:


class-map type inspect match-all CM_OUTSIDE_INSIDE
  match access-group name OUTSIDE_TO_INSIDE
  match class-map CM_INSPECTED_PROTOCOLS
class-map type inspect match-all CM_INSIDE_OUTSIDE
  match access-group name INSIDE_TO_OUTSIDE
  match class-map CM_INSPECTED_PROTOCOLS


To meet the packet permit-log requirement, we need a “parameter-map” that will be applied in the policy-map that references the previous class-maps:


parameter-map type inspect PARAM_AUDIT_LOG
  audit-trail on


Next, the class-maps and parameter-map are referenced in a pair of policy-maps:


policy-map type inspect PM_ZBFW_OUTSIDE_INSIDE
  class type inspect CM_OUTSIDE_INSIDE
    inspect PARAM_AUDIT_LOG
  class class-default
    drop log
policy-map type inspect PM_ZBFW_INSIDE_OUTSIDE
  class type inspect CM_INSIDE_OUTSIDE
    inspect PARAM_AUDIT_LOG
  class class-default
    drop log


Note the “drop log” statement in the final section. Similar to the implicit deny rule in ACLs, ZBFW policies include a class-default with a drop statement. If you want packet drops to be logged, however, you need to explicitly add the “log” parameter to the drop command.

At this point, the policies are configured. Now they need to be linked to interfaces and traffic direction. This is where the “zones” come in:


zone security INSIDE
  description to R6
zone security OUTSIDE
  description to R4
interface FastEthernet0/0
  zone-member security OUTSIDE
interface FastEthernet0/1
  zone-member security INSIDE


Finally, I created zone pairs that associate zones, traffic direction, and traffic policy:


zone-pair security ZP_OUTSIDE_INSIDE source OUTSIDE destination INSIDE
  service-policy type inspect PM_ZBFW_OUTSIDE_INSIDE
zone-pair security ZP_INSIDE_OUTSIDE source INSIDE destination OUTSIDE
  service-policy type inspect PM_ZBFW_INSIDE_OUTSIDE


At this point, the zone-based firewall should be working and ready to test. Based on the policy defined above, traffic from R4’s loopback address should be able to reach R6’s loopback address, but traffic from other interfaces on R4 should be dropped:


R4#ping 6.6.6.6 source 4.4.4.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
Packet sent with a source address of 4.4.4.4
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R4#
R4#ping 6.6.6.6 source 1.1.45.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
Packet sent with a source address of 1.1.45.4
.....
Success rate is 0 percent (0/5)


So far, this looks good. The log on R5 should verify the results above:


*Jun 10 14:29:02.604: %FW-6-SESS_AUDIT_TRAIL_START: (target:class)-(ZP_OUTSIDE_INSIDE:CM_OUTSIDE_INSIDE):Start icmp session: initiator (4.4.4.4:0) -- responder (6.6.6.6:0)
*Jun 10 14:29:13.100: %FW-6-SESS_AUDIT_TRAIL: (target:class)-(ZP_OUTSIDE_INSIDE:CM_OUTSIDE_INSIDE):Stop icmp session: in
itiator (4.4.4.4:8) sent 360 bytes -- responder (6.6.6.6:0) sent 360 bytes

*Jun 10 14:29:16.420: %FW-6-DROP_PKT: Dropping icmp session 1.1.45.4:0 6.6.6.6:0 on zone-pair ZP_OUTSIDE_INSIDE class class-default due to  DROP action found in policy-map with ip ident 0
*Jun 10 14:29:19.812: %FW-6-LOG_SUMMARY: 2 packets were dropped from 1.1.45.4:8 => 6.6.6.6:0 (target:class)-(ZP_OUTSIDE_INSIDE:class-default)


The first two lines show the permitted session; the second pair of lines show the dropped session. I haven’t had the chance yet to examine log entries for traffic types other than TCP/UDP/ICMP, but at first glance it looks like the log entries are formatted in a way that’s friendly to machine parsing.

My next step should probably be to get the Cisco Press eBook on ZBFW by the formidable Ivan Pepelnjak, which I’m a little embarassed about not having read.

Links of Interest

I decided to start a periodic link post, mainly to keep track of tech stuff I might want to reference again later.

HTTP Load Testing

Website Security for Webmasters

Reading List on Bayesian Methods

JavaScript PC Emulator < Run VMs in your browser?! We live in the future!

Machine Learning: A Love Story a talk by Hilary Mason

Reverse DNS Lookups with Python

Update 12/22/2014 -- Today I would recommend using the dnspython module instead of PyDNS.

I am neither a professional programmer nor a Python expert, but it's currently my language of choice for quick log parsing projects. I couldn't find an example of how to do reverse DNS queries from the PyDNS library, so I figured I'd post my solution here.

First, install the PyDNS library. Python has some native DNS stuff, but PyDNS just seems a lot nicer overall.

Here's my code to do a PTR query, looking up a IPv4 address to find the corresponding DNS name. This code implements a global dictionary of addresses that have been previously resolved, avoiding the need to query the server repeatedly for the same address. You could remove that part if you're feeding a set of unique addresses into the function.

import  DNS

SERVER = '8.8.8.8' # put your DNS server address here

global ptr_cache

ptr_cache = {}

def get_ptr(address):
    # check cache.    
    if ptr_cache.has_key(address):
         return ptr_cache[address]
   
    #reverse fields in IP address for use with in-addr.arpa query
    fields = address.split('.')
    fields.reverse()
    flippedaddr = '.'.join(fields)

    #query DNS
    d = DNS.DnsRequest(server=DNS_SERVER,timeout=1)

    try:
        r = d.req(flippedaddr+'.in-addr.arpa',qtype='PTR')

    except:
        return "DNS Error"
   
    name = r.answers[0]['data']
    if name:
        ptr_cache[address] = name
    return name

Converting text to number in Excel

It took me too long to figure out how to typecast a string to a number in Excel. Preserving it here for posterity; use the "=VALUE()" function.

Find Active Hostnames Per Network

Here's a quick trick I use to find the hostnames of all active IPv4 devices in a subnet:

$ ssh routerIP.test.local 'sh arp | i Vlan70' | awk '{print $2}' | xargs -i dig -x {} +short

Password:

foo-tstsrv-01.test.local.
foo-gissrv-01.test.local.
foo-appsrv-01.test.local.
foo-filsrv-03.test.local.
foo-filsrv-02.test.local.

Translated into English:

1. ssh routerIP.test.local 'sh arp | i Vlan70' displays the ARP table for Vlan 70 on the router acting as the default gateway for that VLAN.
2. awk '{print $2}' extracts the second field from the output, which is the IPv4 address for the ARP entry.
3. xargs -i dig -x {} +short takes each one of those IPv4 addresses and queries DNS for the hostname associated with the IP address (that is, the PTR record), using the "dig -x" command, with the +short parameter to display only the hostname. The {} syntax is a part of the xargs command which causes the output from the previous command (that is, the awk command output which produces just an IPv4 address) to be inserted in the place of the {} characters.

To run this on Windows, you need to have both Cygwin and the dig command installed.

Troubleshoot Your Corporate-Speak

A friend of mine recently told me that I "have a hang-up about the meanings of words".

Guilty as charged. I just read yet another press release that uses the idiotic expression "best-of-breed"--I'm at the point where seeing that phrase makes me want to throw a rock at the monitor.

Here's a simple test for a CorporateSpeak buzzphrase: could you say the opposite, without sounding like a crazy person? If not, then your buzzphrase is meaningless. For example:

"Best-of-Breed Vendors Offer Tested and Validated Solutions for Multiple Cisco VXI Deployment Options"

Now, try the opposite:

"Mediocre Vendors With More Successful Competitors Offer Tested and Validated Solutions for Multiple Cisco VXI Deployment Options"

or

"Worst-of-Breed Vendors Offer Tested and Validated Solutions for Multiple Cisco VXI Deployment Options"

No sane person would write that. Thus, the result of the test is that the changed phrase, "best-of-breed", obscures rather than enhances meaning.

Another one I hear all the time is "IT should work to serve the needs of the business." This one doesn't have any suspicious buzzphrases in it, but it's still completely meaningless: can you imagine saying "IT should not work to serve the needs of the business"? Of course not; you'd sound insane. Compare that with a similar, but meaningful and concrete sentence: "IT should work to reduce costs by improving the performance of the accounting servers." With this sentence, IT is still "serving the needs of the business", but you could clearly state the opposite and still have meaning: one could certainly argue that IT's efforts are better spent in areas other than accounting without sounding crazy.

One final example from a friend at a software firm. He received this in email from a guy in sales:

"We need to write software that customers want to buy."

The utter poverty of meaning in that waste of bits is left as an exercise for the reader.

Now, I realize that these sorts of expressions have purposes other than enhancing meaning: they might serve to solicit agreement from the reader as a prelude to a more controversial assertion, or they might simply be not-so-subtle attempts at marketing tricking the reader into a positive first impression. I don't really accept those excuses, though: rational people seek to create meaning, not to obscure it. Get into the habit of troubleshooting your meaning. One way is to test the opposite.

Endnote:

I have no idea if this idea is original or not. It seems like a simple enough idea that I may have gotten it from someone else, but if so, I don't remember and can't attribute the source.

Generate DNS Import from Solarwinds Orion NCM

It's nice to have entries in your internal DNS for router interface names; it makes stuff like traceroute a whole lot easier to read. Many small-to-mid-size companies use Solarwinds products for network management.

This perl script takes the output that you get from using Solarwinds Network Configuration Manager to run the "show ip interface brief | exclude unassigned" command on a group of Cisco IOS devices and massages it into a format that looks like this:

hostname-interfaceName-interfaceNumber a.b.c.d

where a.b.c.d is the IP address assigned to the interface. This format is suitable for import into many DNS servers.

This code doesn't attempt to abbreviate interface names at all, but it could easily be modified to do so.

I run this on Windows using Cygwin and it works fine. The only caveat is that you need to have the dig command installed.

#!/usr/bin/perl
#
# takes output from a Solarwinds Orion NCM command script
# that runs "show ip interface brief | exclude unassigned"
# and produces a "hostname IP" output for import into DNS
# this makes traceroutes more readable
#
# usage: ./interface2dns.pl < inputfile.txt
# inputfile.txt will look like this
# when produced by Orion NCM 6.x:
#
# routerA.test.com  (10.38.16.126)
# Interface   IP-Address      OK? Method Status Protocol   
# Loopback0   172.29.255.1    YES NVRAM  up     up     
# Vlan163     10.38.16.126    YES NVRAM  up     up     
# Vlan165     10.38.16.117    YES NVRAM  up     up  
#
#
while (<>){
   
    chomp; # remove newline characters

# find the line with the hostname in it by searching
# for the "(" character
if ($_=~/\(/) {

    # split that line on the . character
    @hostnameFields = split /\./,$_;
    # hostname is the first element before the first .
    $hostname=$hostnameFields[0];

}

# identify lines with "up" in them to remove garbage
if ($_=~/\s+up\s+/) {

    #split the good lines into space-separated fields
    @fields=split  /\s+/,$_;
   
    #find RFC1918-like addresses only
    if ($fields[1]=~/^10\.|172\.|192\./){
     
        # skip IP addresses that are already in DNS
          unless (`dig -x $fields[1] +short`) {
     
          #substitute - for / and : in dns names
          $fields[0]=~s/\/|:/-/g;
 
          #print in "hostname-interface ipaddr" format
          print "$hostname-$fields[0] $fields[1]\n";    
         }
     }
}
}

Encrypted GRE Tunnel with ASA for Encryption Offload

I have a requirement to connect two internal routers over a high-speed, 3rd party, non-Internet network. The two routers need to run EIGRP with each other. We want to encrypt the link, but the routers don't support IPSec in their current configuration. Purchasing IPSec acceleration hardware for them is expensive, but we happen to have two ASAs in inventory that aren't currently in production.

The simplest solution for this is to connect the two routers with a GRE tunnel, then use the ASAs to encrypt the GRE traffic.

While I've worked with ASAs quite a bit as stateful packet filters and as remote access VPN headends, it's been a really long time since I've used one in a point-to-point VPN. I thought I'd blog about the lab proof-of-concept so that I don't forget everything about the configuration.

My lab topology looks like this:

R4---ASA1---ASA2---R5

R4 and R5 have a standard GRE tunnel configured between them, running EIGRP to advertise their loopbacks. The tunnel destination is statically routed.

On R4:

interface Loopback0
ip address 4.4.4.4 255.255.255.0
!
interface FastEthernet0/0
description link to ASA1
ip address 1.1.1.4 255.255.255.0
!
interface Tunnel50
description GRE tunnel to R5, will be encrypted by ASA1
ip address 50.1.1.4 255.255.255.0
tunnel source FastEthernet0/0
tunnel destination 2.2.2.5
!
ip route 2.2.2.5 255.255.255.255 1.1.1.1
!
router eigrp 1
network 4.0.0.0
network 50.1.1.0 0.0.0.255

On R5:

interface Loopback0
ip address 5.5.5.5 255.255.255.0
!
interface FastEthernet0/0
description link to ASA2
ip address 2.2.2.5 255.255.255.0
!
interface Tunnel50
description GRE tunnel to R4, will be encrypted by ASA2
ip address 50.1.1.5 255.255.255.0
tunnel source FastEthernet0/0
tunnel destination 1.1.1.4
!
router eigrp 1
network 5.0.0.0
network 50.1.1.0 0.0.0.255
no auto-summary

[Note: the difference in the EIGRP configs isn't a mistake. The lab routers are running two different images, one of which has auto-summary disabled by default. The other has it enabled by default, so I had to explicitly turn it off.]

This is a pretty standard GRE configuration that is used all the time to make a virtual point-to-point circuit across any other network. I'm intentionally leaving out the MTU complications for now.

Here's the interface configuration for ASA1:

interface Ethernet0/2
description link to R4
nameif inside
security-level 100
ip address 1.1.1.1 255.255.255.0
!
interface Ethernet0/1
description link to ASA2 representing 3rd party network
nameif outside
security-level 0
ip address 100.1.1.1 255.255.255.0

and here's the same configuration from ASA2:

interface Ethernet0/2
description link to R5
nameif inside
security-level 100
ip address 2.2.2.1 255.255.255.0
!
interface Ethernet0/1
description link to ASA1 representing 3rd party network
nameif outside
security-level 0
ip address 100.1.1.2 255.255.255.0

Next, we need to configure IPSec on the ASAs. This is pretty similar to doing the same thing on a IOS router, with a couple of differences:
crypto ipsec transform-set ESP_3DES esp-3des esp-sha-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto map P2P_CRYPTO_CM 10 match address R5_PHYSICAL
crypto map P2P_CRYPTO_CM 10 set peer 100.1.1.1
crypto map P2P_CRYPTO_CM 10 set transform-set ESP_3DES
crypto map P2P_CRYPTO_CM interface outside
crypto isakmp enable outside
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
!
access-list R5_PHYSICAL extended permit ip host 2.2.2.5 host 1.1.1.4

When I first set this up, I forgot the unfamiliar crypto isakmp enable outside command, and it took me a few minutes of looking at debugs to figure out why the ASA was dropping the IKE Phase 1 packets.

The other part that's different from the IOS configuration is the presence of a "tunnel-group" that defines the tunnel type and the IKE pre-shared key:

tunnel-group 100.1.1.1 type ipsec-l2l
tunnel-group 100.1.1.1 ipsec-attributes
pre-shared-key *****

I also needed to turn off NAT control so that the ASA wouldn't drop packets without a pre-defined NAT translation:

no nat-control

The configuration on the other ASA is identical, except that the crypto ACL and peer addresses are reversed, just like they would be in an IOS configuration.

Detecting a Transparent Proxy with Wireshark/Tshark

Recently I got pulled into a debate between two colleagues who were troubleshooting a problem where some users could access a website over SSL, and others couldn't. One person was arguing that the problem was caused by client misconfiguration, and the other was arguing that it wasn't. Following my mantra "when in doubt, capture packets", we captured some traffic and had a look. I'm not going to go through the entire troubleshooting process; rather I'm going to focus on what was ultimately causing the problem. Here's the packet sequence, output from Tshark with some of the TCP details removed to make it fit:

921 13.795492 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [SYN]
926 13.837731 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [SYN, ACK]
927 13.837753 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [ACK]
928 13.838086 10.100.100.192 -> 184.86.133.186 SSL Client Hello
932 13.840253 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [RST]
943 13.879563 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [ACK]
944 13.879588 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [RST]
945 13.880052 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [RST]
946 13.881491 184.86.133.186 -> 10.100.100.192 TLSv1 Server Hello
947 13.881513 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [RST]
948 13.881535 184.86.133.186 -> 10.100.100.192 TLSv1 Certificate, Server Hello Done
949 13.881545 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [RST]
950 13.881554 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [RST]
953 13.882063 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [RST]


The key thing to note here is the delta between the SYN and SYN/ACK: about 42ms. When I was viewing this in Wireshark I set a packet time reference on packet #921, using the "Ctrl T" keyboard shortcut; this makes it easier to see the delta values.

I then set another time reference on packet 931, the TLSv1 Client Hello. Immediately following this, less than 1 millisecond later, we see a RST come back from the server. Red flag! Since we already established the probable latency between the hosts as ~42ms using the SYN - SYN/ACK pair, this is extremely suspicious.

A few packets later, we see a TLSv1 Server Hello message inbound, AFTER the RST. The delta? Approximately 42ms, exactly what we'd expect.

I immediately inferred from this that a transparent proxy content filter was spoofing the RST due to something that it deemed to be objectionable content, and for whatever reason it wasn't notifying the user.

I talked to the administrator for the content filter, and indeed, a policy had been incorrectly applied to some users that blocked the content in question.

Another win for packet capture.

playing with VLAN-based QoS

A friend of mine brought up an interesting question yesterday:

Will the "mls qos vlan-based" command mark packets if the packets aren't routed by the SVI where the service policy is applied?

I had never even considered this before, so I guessed "no" before reading the docs. Then I set it up in the lab, and the answer (superficially) still seemed to be "no". Then, however, I executed the old RTFM maneuver: the docs made it sound like the answer should be "yes". So I thought I'd have a closer look. Here are the details.

The test topology looks like this:

R5---Switch1---Switch2---Cat3750---R6

• R5 & R6 are 2800 series routers.
• Switch1 and Switch2 are 3560s.
• Cat3750, unsurprisingly, is a Catalyst 3750, and the subject of the test.

All of the inter-switch links are 802.1q trunks, and the two routers are connected to access ports in VLAN 6.

• R5's interface IP address is 10.6.6.5, with a loopback of 5.5.5.5.
• R6's interface IP address is 10.6.6.6, with a loopback of 6.6.6.6.
• R5 and R6 are running OSPF on all interfaces and are neighbors when the test begins.

First, I set up a QoS policy on the 3750 that would make packets distinguishable from each other:

ip access-list extended ALL
permit ip any any
!
class-map match-any FOO
match access-group name ALL
!
policy-map IN-MARKING
class FOO
set ip dscp cs2
class class-default
set dscp cs1

Then I applied the policy-map to an unrouted SVI for VLAN 6:

interface Vlan6
no ip address
service-policy input IN-MARKING

Then I set R6's access port for VLAN-based QoS:

interface FastEthernet1/0/2
switchport access vlan 6
switchport mode access
mls qos vlan-based
spanning-tree portfast

Since all traffic should be marked as CS2, the OSPF hello packets being exchanged between R5 and R6 should get marked immediately if the policy is working, but I also generated some ICMP and telnet traffic just for good measure.

My superficial, pre-RTFM diagnosis, said it wasn't working:

c3750#sh policy-map interface
Vlan6

Service-policy input: IN-MARKING

Class-map: FOO (match-any)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: access-group name ALL
0 packets, 0 bytes
5 minute rate 0 bps

Class-map: class-default (match-any)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: any
0 packets, 0 bytes
5 minute rate 0 bps

After my RTFM episode, I started wondering if the packets were getting marked anyway, but not counted by the switch. Idecided to try applying a service policy on R5 that would have rather dramatic effects if the packets were in fact marked:

class-map match-any CM_CS2
match ip dscp cs2
!
policy-map PM_DROP_CS2
class CM_CS2
drop

then...

R5(config)#int f0/0
R5(config-if)#service-policy input PM_DROP_CS2
R5(config-if)#
R5(config-if)#
*Nov 18 15:37:18.097: %OSPF-5-ADJCHG: Process 1, Nbr 20.1.1.6 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Dead timer expired

R5#sh policy-map interface f0/0 input class CM_CS2
FastEthernet0/0

Service-policy input: PM_DROP_CS2

Class-map: CM_CS2 (match-any)
20 packets, 1880 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: ip dscp cs2 (16)
20 packets, 1880 bytes
5 minute rate 0 bps
drop

So... in conclusion: the "mls qos vlan-based" command does indeed work for service policies applied to interfaces that don't route packets for the VLAN... essentially, the command causes the access port to inherit the QoS policy applied to the SVI, and the SVI doesn't count them. This does actually make sense, but it took working through the lab to make my brain process it correctly.

Links to Network World Blog Posts

Last spring I wrote a series of guest blog posts on Network World, so I thought I'd link them here for future reference:

IOS Output Filters
More IOS Output Filters
Finding IP Addresses
NetFlow Part 1
NetFlow Part 2
NetFlow Part 3
When in Doubt, Capture Packets

Counting Regex Matches on Catalyst Switches

As an old-time Unix/Linux guy, I’ve been wanting the IOS equivalent of the “wc -l” command for oh, about a million years now. Juniper has had this in JunOS for quite some time, via their “count” command. In Unix/Linux shell environments, “wc -l” counts the number of lines in the input. This is very useful for things like counting the number of unused ports on a switch. Until recently, I was stuck with using SSH one-liners to pipe IOS commands through a shell filter, like this:

$ ssh 10.1.1.2 'sh int status | i not|disa' | wc -l
Password:
19

In English: this executes the command “show interface status | include not|disa” via SSH on the remote device, which returns all the lines that are in either the “notconnected” or “disabled” status. Then it feeds those lines to the “wc -l” command, which counts the lines.

A couple of months ago, however, I noticed that recent versions of IOS for the Catalyst access switches (3560/3750, and probably the 2960--I haven’t tested the latter) have a “count” filter:

Switch#sh int status | count ?
LINE Regular Expression

Switch#sh int status | count not|disa
Number of lines which match regexp = 19

Anyway, nothing particularly groundbreaking here, just a nice feature that’s long overdue--I hope it gets ported into other IOS images as well.

Cisco Live 2010

This was my 6th consecutive year at CiscoLive (formerly known as Networkers), and as always I had a great time. I thought I'd write down a few brief thoughts on my sessions and other experiences this year.

802.1x 8-Hour Techtorial
This was the first time I've done a "techtorial" (Cisco's term for a 4 or 8 hour expanded seminar that costs extra) since 2006, and it was probably the best one I've done. All of the presenters were excellent, and they did a great job of keeping the class engaged by switching frequently between lecture and live demonstrations by three different instructors. They also included a real-world case study, presented by the actual customer involved (a large Canadian university). Cisco has a tendency to make up artificial case studies (or anonymize them to the point of making them pointless), so it was great to see a live customer on stage, presenting the entire implementation process, warts and all.

LISP - A Next Generation Networking Architecture
I have read a fair bit about LISP over the last couple of years, but this was the first time I've gone to a session on it. Basically, Dino and company are trying to solve three or four of the biggest problems in networking in one fell swoop: 1) global routing table size, 2) certain types of IPv4/IPv6 transport issues, 3) virtual machine mobility, and possibly 4) other mobility problems. I really don't have the background to evaluate a protocol that's designed to solve extremely difficult problems at a global scale, but it was fascinating to see the thought process and design issues involved.

Routed Fast Convergence and High Availability
I have been hearing about this session for years, and it didn't disappoint. I was already familiar with most of the tools discussed, but the devil is in the details, and I came away with a much better understanding of many techniques used to achieve fast convergence in routed networks.

Smart Grid: Developing a Communications Architecture for the Utility of the Future
I didn't intend to visit this session initially, but the speaker was late for my scheduled session, and I had no interest in sitting around waiting for him to show up. This session was happening nearby, and I knew that Bill Parkhurst is something of a network architecture guru, so it was an easy pick. Even though I have absolutely no background in the electrical utility world, this was a very useful session from a general professional development perspective.

Unified HA Network Design: The Evolution of the Next Generation Network
If I could recommend only one session on large-scale network design, this would be it. These guys are working on the largest, most failure-sensitive networks on the planet, and they're giving away what they've learned. What more needs to be said?

Advanced Security Management & Incident Response
This was my second time attending this session (the last time was in 2007), and it was definitely worth attending a second time. I really like operationally-focused sessions (as opposed to product-focused ones), and that's what this one is all about. The presenters are front-line senior incident responders in Cisco's internal security organization, and it's great to see how they deploy Cisco tools and even (gasp) non-Cisco tools to respond to actual security incidents. I really hope they can convince the powers-that-be to let them run this as an expanded, 8-hour techtorial on security operations.

Those are the highlights of my sessions. I didn't attend a single session this year that was actually bad, but those were the ones that stood out the most.

Other thoughts: the meals were above average compared to previous years, except for breakfast. I hate getting to a breakfast and seeing nothing but bread products and some random, lonely looking fruit. The CCIE party was great; probably the best one I've attended. The CCIE NetVet reception with John Chambers was also excellent. My biggest complaint continues to be the ominous warnings about a $100 fee to replace a lost conference badge. I've never lost mine, but this just seems patently ridiculous.

As always, though, simply meeting other networking-focused professionals and renewing friendships with people I've known from previous years was the best part of the show.

Hoping to be back next year!