Loopback Mountain

Cisco CDR Timestamp and IP Address Conversion

2012-02-10T21:51:00.003-07:00

The call detail records (CDRs) used by Cisco Unified Communications Manager (aka Call Manager) can be tricky to interpret. Two fields that are frequently confusing are the IP address fields and the timestamp field.

Continuing my theme of using Python to help with Cisco product administration, here's how to convert timestamps and IP addresses to human-readable format.

Timestamps are pretty easy. They're recorded in UNIX epoch time and there are lots of examples available showing how to convert them using Excel or various scripting tools. In Python:

import time

def time_to_string(time_value):

    """convert Unix epoch time to a human readable string"""

    return time.strftime("%m/%d/%Y %H:%M:%S",time.localtime(float(time_value)))

It took me a while to figure out how to convert IPv4 addresses, since CUCM uses signed 32-bit integers to represent IP addresses and Python's long integers can be of infinite length. First, a review of how to do the conversion manually is in order. Let's say that a CUCM CDR lists an IP address as "-2126438902". First, we convert this signed 32-bit integer to hex using Windows calculator:

-2126438902 = 0x81411E0A

Next, we break the hex number into 1-byte chunks, and reverse the order:

0x0A

0x1E

0x41

0x81

Finally, we convert each byte to decimal and put them together into an IP address:

10.30.65.129

Here's the Python function to do the conversion:

And here it is in the interpreter showing that it works for reasonable input types:

>>> int_to_ip('-2126438902')

'10.30.65.129'

>>> int_to_ip(-2126438902)

'10.30.65.129'

>>> int_to_ip(-2126438902L)

'10.30.65.129'

I don't have access to an IPv6-aware CUCM install, so you're on your own for that!

Amusement: Python and Netmasks

2012-01-29T14:20:00.001-07:00

As a network engineer, it's not uncommon for me to need to convert between hex and decimal. While I'm reasonably good at doing this in my head for smaller numbers, when it comes to deciphering stuff like higher TCP or UDP port numbers written in hex, I usually end up using the Python interpreter that's usually open somewhere on my machine. For me, the Python interpreter is the best general purpose calculator app I've found. Using the port number for Flash as an example:

jswan$ python

Python 2.7.2 (v2.7.2:8527427914a2, Jun 11 2011, 15:22:34) 

[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> 0x78f

1935

>>> hex(1935)

'0x78f'

The topic of numeric base conversions often makes my mind drift to every former networking instructor's pet topics, IPv4 subnetting. The other day, I started playing with using the Python interpreter to find network IDs, which is really easy as long as you're using hex:

>>> hex(0x0a0a0a25 & 0xffffffe0)

'0xa0a0a20'

This little snippet finds the bitwise-AND of 10.10.10.37 and 255.255.255.224, which as every up-and-coming CCNA knows is the network ID for that subnet: 10.10.10.32. Back when I was teaching Cisco classes full-time, I used to have a never-ending argument with another instructor about the fact that I didn't teach bitwise operations as part of subnetting: my position was that if all you are doing is solving subnet problems with your squishy human brain, you don't need to learn a bunch of truth tables when there are easier ways to do it in human memory. However, if you're writing code you actually do need to do bitwise operations.

Unfortunately most of us (me included) aren't wired for reading IPv4 addresses in hex. So I started wondering how little Python code I could use to calculate network IDs for IPv4 in dotted decimal. A little screwing around and I started wondering if I could fit the entire thing into a Twitter post. Here's what I came up with:

while 1:'.'.join([str(a&m)for a,m in zip([int(n)for n in raw_input('addr?').split('.')],[int(n)for n in raw_input('mask?').split('.')])])

Try it:

addr?1.1.1.37

mask?255.255.255.224

'1.1.1.32'

Now I realize this bit of code is neither particularly clever nor easy to read, which makes it bad code. But hey, it fits in a single tweet. It works by using one of Python's coolest features, the list comprehension. If we start with the innermost parts, it makes more sense:

[int(n) for n in raw_input('addr?').split('.')]

[int(n) for n in raw_input('addr?').split('.')]

These two sections return lists of integers corresponding to the address and mask the user enters. For example:

[1,1,1,37]

[255,255,255,224]

Next, the "zip" function returns a list of tuples that pair corresponding entries in the two lists:

[(1, 255), (1, 255), (1, 255), (37, 224)]

Next, the outermost list comprehension performs a bitwise-AND of each tuple, returning the octets of our network ID in a new list:

[1,1,1,32]

Finally, the "join" method puts them back together into "1.1.1.32", and "while 1:" makes it a loop until you ctrl-c out of it.

Working with netmasks in CIDR notation is a bit more complicated, and requires more than one line of code--I'll save that for another post.

Find Unique Interface Configs on Switches

2011-12-21T16:58:00.001-07:00

Recently I needed to find all of the interfaces with non-standard configurations on a bunch of Catalyst 3750 switches. I wrote a simple Python script to automate the process. To run this, save it in a directory with all of the configurations you want to check. Then edit the name.startswith() section near the bottom to match the naming convention for your config files; in my case all of my configs start with "c3750". Then run the script from your CLI:

>python dup_conf.py

If you're on a Mac or Linux system, Python should be installed by default. On Windows, you'll need to install it or run Cygwin. Note that in the default version, the script ignores SVIs and Gigabit interfaces; uncomment the appropriate lines to include them.

The output looks like this:

in config file c3750-1A.txt

interfaces

interface FastEthernet1/0/35

configured like

switchport trunk encapsulation dot1q

switchport trunk native vlan 76

switchport mode trunk

switchport voice vlan 65

no logging event link-status

srr-queue bandwidth share 10 10 60 20

srr-queue bandwidth shape 10 0 0 0

mls qos trust device cisco-phone

mls qos trust dscp

auto qos voip cisco-phone 

no mdix auto

spanning-tree portfast

**************************************************

interfaces

interface FastEthernet1/0/11

interface FastEthernet1/0/12

configured like

switchport trunk encapsulation dot1q

switchport trunk native vlan 80

switchport mode trunk

switchport voice vlan 65

srr-queue bandwidth share 10 10 60 20

srr-queue bandwidth shape 10 0 0 0

mls qos trust device cisco-phone

mls qos trust dscp

auto qos voip cisco-phone 

no mdix auto

spanning-tree portfast

**************************************************

interfaces

interface FastEthernet1/0/1

interface FastEthernet1/0/2

interface FastEthernet1/0/3

interface FastEthernet1/0/4

interface FastEthernet1/0/5

interface FastEthernet1/0/6

interface FastEthernet1/0/7

interface FastEthernet1/0/8

interface FastEthernet1/0/9

interface FastEthernet1/0/10

interface FastEthernet1/0/13

interface FastEthernet1/0/14

interface FastEthernet1/0/15

interface FastEthernet1/0/16

configured like

switchport access vlan 64

switchport voice vlan 65

srr-queue bandwidth share 10 10 60 20

srr-queue bandwidth shape 10 0 0 0

mls qos trust device cisco-phone

mls qos trust dscp

auto qos voip cisco-phone 

no mdix auto

spanning-tree portfast

**************************************************

Summarize Router Throughput in Packets/Second

2011-11-10T09:52:00.001-07:00

Here's a quick one-liner to summarize the current packets-per-second throughput on an entire IOS router:

$ ssh 10.1.2.3 'sh int summ' | awk '/^*/{RXPPS+=$8;TXPPS+=$10} END {print "RXPPS=" RXPPS,"TXPPS=" TXPPS}'
jswan@10.1.2.3's password:
RXPPS=1613 TXPPS=1539

Zone Based Firewall Configuration Example

2011-06-10T10:33:00.002-06:00

There aren’t a ton of examples available for IOS Zone-Based Firewall configurations, so I thought I’d put up one with which I’ve been working recently.

My test network looks like this:

R4 ------------------- R5 ----------------------- R6

R4 Loopback: 4.4.4.4

R4 to R5 link: 1.1.45.0/24 

R5 Loopback: 5.5.5.5

R5 to R6 link: 1.1.56.0/24 

R6 Loopback: 6.6.6.6

My goal is to express this policy:

Hosts in group X on the OUTSIDE network can initiate sessions to hosts on the INSIDE network. Return traffic for those sessions should be statefully permitted. In the lab, group X is represented by R4's loopback address.
Any host on the INSIDE network can initiate sessions to hosts in group X on the OUTSIDE network. Return traffic for those sessions should be statefully permitted. In the lab, the inside hosts are represented by R6's loopback address.
Permitted traffic should be logged. All other traffic should be dropped and logged.

The network on which the production version of this lab test will be deployed currently uses extended access-lists to implement the policy. Access-lists have the advantage of being familiar to anyone with a basic knowledge of IOS, but they have a lot of disadvantages:

Hard to read and troubleshoot as they grow.
No stateful awareness. Trying to retrofit statefulness onto extended ACLs to allow return traffic requires ugly hacks with source port restrictions, filtering on ACK bits, etc. This path eventually leads to mind-numbing troubleshooting problems, and is less than optimal in its security.
Extended ACLs don’t have application-layer awareness.

There are several tools in IOS to facilitate stateful traffic inspection, but Zone-Based Firewalls are the newest and most flexible, so it made sense to use them.

Most of the examples I found via Google show the simple case of inside hosts having unfettered access to the outside, and outside hosts having no access to the inside--a classic stateful firewall design. My case is only slightly more complex, but it still took me a couple of tries to make it work.

First, I built access-lists defining the traffic to be allowed. I decided to use the relatively new IOS object-group support to make it easier to add and delete hosts on the respective networks:

object-group network OG_INSIDE_HOSTS
host 6.6.6.6
object-group network OG_OUTSIDE_ALLOWED
host 4.4.4.4
ip access-list extended INSIDE_TO_OUTSIDE
permit ip object-group OG_INSIDE_HOSTS object-group OG_OUTSIDE_ALLOWED
ip access-list extended OUTSIDE_TO_INSIDE
permit ip object-group OG_OUTSIDE_ALLOWED object-group OG_INSIDE_HOSTS

You might notice that there’s no “deny ip any any log” statement in these ACLs, which is strange given requirement #3 above. It turns out that when using ZBFW, you configure drop logging elsewhere; I’ll cover that below.

I couldn’t find a way to express protocol inspection policy and IPv4 address policy in the same class-map, so I had to use a hierarchical configuration. To express the protocol inspection policy, I built a class-map to define the layer 4 protocols that will be inspected by the firewall:

class-map type inspect match-any CM_INSPECTED_PROTOCOLS
match protocol icmp
match protocol tcp
match protocol udp

This class-map does simple generic TCP/UDP/ICMP inspection, but it could easily be extended or rewritten to use much more complex inspection rules.

Next, I built class-maps for each direction that match the appropriate ACL and the inspection policy configured above:

class-map type inspect match-all CM_OUTSIDE_INSIDE
match access-group name OUTSIDE_TO_INSIDE
match class-map CM_INSPECTED_PROTOCOLS
class-map type inspect match-all CM_INSIDE_OUTSIDE
match access-group name INSIDE_TO_OUTSIDE
match class-map CM_INSPECTED_PROTOCOLS

To meet the packet permit-log requirement, we need a “parameter-map” that will be applied in the policy-map that references the previous class-maps:

parameter-map type inspect PARAM_AUDIT_LOG
audit-trail on

Next, the class-maps and parameter-map are referenced in a pair of policy-maps:

policy-map type inspect PM_ZBFW_OUTSIDE_INSIDE
class type inspect CM_OUTSIDE_INSIDE
    inspect PARAM_AUDIT_LOG
class class-default
    drop log
policy-map type inspect PM_ZBFW_INSIDE_OUTSIDE
class type inspect CM_INSIDE_OUTSIDE
    inspect PARAM_AUDIT_LOG
class class-default
    drop log

Note the “drop log” statement in the final section. Similar to the implicit deny rule in ACLs, ZBFW policies include a class-default with a drop statement. If you want packet drops to be logged, however, you need to explicitly add the “log” parameter to the drop command.

At this point, the policies are configured. Now they need to be linked to interfaces and traffic direction. This is where the “zones” come in:

zone security INSIDE
description to R6
zone security OUTSIDE
description to R4
interface FastEthernet0/0
zone-member security OUTSIDE
interface FastEthernet0/1
zone-member security INSIDE

Finally, I created zone pairs that associate zones, traffic direction, and traffic policy:

zone-pair security ZP_OUTSIDE_INSIDE source OUTSIDE destination INSIDE
service-policy type inspect PM_ZBFW_OUTSIDE_INSIDE
zone-pair security ZP_INSIDE_OUTSIDE source INSIDE destination OUTSIDE
service-policy type inspect PM_ZBFW_INSIDE_OUTSIDE

At this point, the zone-based firewall should be working and ready to test. Based on the policy defined above, traffic from R4’s loopback address should be able to reach R6’s loopback address, but traffic from other interfaces on R4 should be dropped:

R4#ping 6.6.6.6 source 4.4.4.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
Packet sent with a source address of 4.4.4.4
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R4#
R4#ping 6.6.6.6 source 1.1.45.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
Packet sent with a source address of 1.1.45.4
.....
Success rate is 0 percent (0/5)

So far, this looks good. The log on R5 should verify the results above:

*Jun 10 14:29:02.604: %FW-6-SESS_AUDIT_TRAIL_START: (target:class)-(ZP_OUTSIDE_INSIDE:CM_OUTSIDE_INSIDE):Start icmp session: initiator (4.4.4.4:0) -- responder (6.6.6.6:0)
*Jun 10 14:29:13.100: %FW-6-SESS_AUDIT_TRAIL: (target:class)-(ZP_OUTSIDE_INSIDE:CM_OUTSIDE_INSIDE):Stop icmp session: in
itiator (4.4.4.4:8) sent 360 bytes -- responder (6.6.6.6:0) sent 360 bytes

*Jun 10 14:29:16.420: %FW-6-DROP_PKT: Dropping icmp session 1.1.45.4:0 6.6.6.6:0 on zone-pair ZP_OUTSIDE_INSIDE class class-default due to DROP action found in policy-map with ip ident 0
*Jun 10 14:29:19.812: %FW-6-LOG_SUMMARY: 2 packets were dropped from 1.1.45.4:8 => 6.6.6.6:0 (target:class)-(ZP_OUTSIDE_INSIDE:class-default)

The first two lines show the permitted session; the second pair of lines show the dropped session. I haven’t had the chance yet to examine log entries for traffic types other than TCP/UDP/ICMP, but at first glance it looks like the log entries are formatted in a way that’s friendly to machine parsing.

My next step should probably be to get the Cisco Press eBook on ZBFW by the formidable Ivan Pepelnjak, which I’m a little embarassed about not having read.

Links of Interest

2011-05-22T20:09:00.000-06:00

I decided to start a periodic link post, mainly to keep track of tech stuff I might want to reference again later.

HTTP Load Testing

Website Security for Webmasters

Reading List on Bayesian Methods

JavaScript PC Emulator < Run VMs in your browser?! We live in the future!

Machine Learning: A Love Story a talk by Hilary Mason

Reverse DNS Lookups with Python

2011-05-13T12:58:00.000-06:00

I am neither a professional programmer nor a Python expert, but it's currently my language of choice for quick log parsing projects. I couldn't find an example of how to do reverse DNS queries from the PyDNS library, so I figured I'd post my solution here.

First, install the PyDNS library. Python has some native DNS stuff, but PyDNS just seems a lot nicer overall.

Here's my code to do a PTR query, looking up a IPv4 address to find the corresponding DNS name. This code implements a global dictionary of addresses that have been previously resolved, avoiding the need to query the server repeatedly for the same address. You could remove that part if you're feeding a set of unique addresses into the function.

import  DNS

SERVER = '8.8.8.8' # put your DNS server address here

global ptr_cache

ptr_cache = {}

def get_ptr(address):
    # check cache.     
    if ptr_cache.has_key(address):
         return ptr_cache[address]
    
    #reverse fields in IP address for use with in-addr.arpa query
    fields = address.split('.')
    fields.reverse()
    flippedaddr = '.'.join(fields)

    #query DNS
    d = DNS.DnsRequest(server=DNS_SERVER,timeout=1)

    try:
        r = d.req(flippedaddr+'.in-addr.arpa',qtype='PTR')

    except:
        return "DNS Error"
    
    name = r.answers[0]['data']
    if name:
        ptr_cache[address] = name
    return name

Converting text to number in Excel

2011-04-08T16:47:00.000-06:00

It took me too long to figure out how to typecast a string to a number in Excel. Preserving it here for posterity; use the "=VALUE()" function.

Find Active Hostnames Per Network

2011-01-12T08:10:00.002-07:00

Here's a quick trick I use to find the hostnames of all active IPv4 devices in a subnet:

$ ssh routerIP.test.local 'sh arp | i Vlan70' | awk '{print $2}' | xargs -i dig -x {} +short

Password:

foo-tstsrv-01.test.local.
foo-gissrv-01.test.local.
foo-appsrv-01.test.local.
foo-filsrv-03.test.local.
foo-filsrv-02.test.local.

Translated into English:

ssh routerIP.test.local 'sh arp | i Vlan70' displays the ARP table for Vlan 70 on the router acting as the default gateway for that VLAN.
awk '{print $2}' extracts the second field from the output, which is the IPv4 address for the ARP entry.
xargs -i dig -x {} +short takes each one of those IPv4 addresses and queries DNS for the hostname associated with the IP address (that is, the PTR record), using the "dig -x" command, with the +short parameter to display only the hostname. The {} syntax is a part of the xargs command which causes the output from the previous command (that is, the awk command output which produces just an IPv4 address) to be inserted in the place of the {} characters.

To run this on Windows, you need to have both Cygwin and the dig command installed.

Troubleshoot Your Corporate-Speak

2011-01-04T14:38:00.003-07:00

A friend of mine recently told me that I "have a hang-up about the meanings of words".

Guilty as charged. I just read yet another press release that uses the idiotic expression "best-of-breed"--I'm at the point where seeing that phrase makes me want to throw a rock at the monitor.

Here's a simple test for a CorporateSpeak buzzphrase: could you say the opposite, without sounding like a crazy person? If not, then your buzzphrase is meaningless. For example:

"Best-of-Breed Vendors Offer Tested and Validated Solutions for Multiple Cisco VXI Deployment Options"

Now, try the opposite:

"Mediocre Vendors With More Successful Competitors Offer Tested and Validated Solutions for Multiple Cisco VXI Deployment Options"

"Worst-of-Breed Vendors Offer Tested and Validated Solutions for Multiple Cisco VXI Deployment Options"

No sane person would write that. Thus, the result of the test is that the changed phrase, "best-of-breed", obscures rather than enhances meaning.

Another one I hear all the time is "IT should work to serve the needs of the business." This one doesn't have any suspicious buzzphrases in it, but it's still completely meaningless: can you imagine saying "IT should not work to serve the needs of the business"? Of course not; you'd sound insane. Compare that with a similar, but meaningful and concrete sentence: "IT should work to reduce costs by improving the performance of the accounting servers." With this sentence, IT is still "serving the needs of the business", but you could clearly state the opposite and still have meaning: one could certainly argue that IT's efforts are better spent in areas other than accounting without sounding crazy.

One final example from a friend at a software firm. He received this in email from a guy in sales:

"We need to write software that customers want to buy."

The utter poverty of meaning in that waste of bits is left as an exercise for the reader.

Now, I realize that these sorts of expressions have purposes other than enhancing meaning: they might serve to solicit agreement from the reader as a prelude to a more controversial assertion, or they might simply be not-so-subtle attempts at ~~marketing~~ tricking the reader into a positive first impression. I don't really accept those excuses, though: rational people seek to create meaning, not to obscure it. Get into the habit of troubleshooting your meaning. One way is to test the opposite.

Endnote:

I have no idea if this idea is original or not. It seems like a simple enough idea that I may have gotten it from someone else, but if so, I don't remember and can't attribute the source.

Generate DNS Import from Solarwinds Orion NCM

2010-12-17T16:09:00.002-07:00

It's nice to have entries in your internal DNS for router interface names; it makes stuff like traceroute a whole lot easier to read. Many small-to-mid-size companies use Solarwinds products for network management.

This perl script takes the output that you get from using Solarwinds Network Configuration Manager to run the "show ip interface brief | exclude unassigned" command on a group of Cisco IOS devices and massages it into a format that looks like this:

hostname-interfaceName-interfaceNumber a.b.c.d

where a.b.c.d is the IP address assigned to the interface. This format is suitable for import into many DNS servers.

This code doesn't attempt to abbreviate interface names at all, but it could easily be modified to do so.

I run this on Windows using Cygwin and it works fine. The only caveat is that you need to have the dig command installed.

#!/usr/bin/perl
#
# takes output from a Solarwinds Orion NCM command script
# that runs "show ip interface brief | exclude unassigned"
# and produces a "hostname IP" output for import into DNS
# this makes traceroutes more readable
#
# usage: ./interface2dns.pl < inputfile.txt
# inputfile.txt will look like this
# when produced by Orion NCM 6.x:
#
# routerA.test.com (10.38.16.126)
# Interface   IP-Address      OK? Method Status Protocol
# Loopback0   172.29.255.1    YES NVRAM up     up
# Vlan163     10.38.16.126    YES NVRAM up     up
# Vlan165     10.38.16.117    YES NVRAM up     up
#
#
while (<>){

    chomp; # remove newline characters

# find the line with the hostname in it by searching
# for the "(" character
if ($_=~/\(/) {

    # split that line on the . character
    @hostnameFields = split /\./,$_;
    # hostname is the first element before the first .
    $hostname=$hostnameFields[0];

}

# identify lines with "up" in them to remove garbage
if ($_=~/\s+up\s+/) {

    #split the good lines into space-separated fields
    @fields=split /\s+/,$_;

    #find RFC1918-like addresses only
    if ($fields[1]=~/^10\.|172\.|192\./){

        # skip IP addresses that are already in DNS
        unless (`dig -x $fields[1] +short`) {

        #substitute - for / and : in dns names
        $fields[0]=~s/\/|:/-/g;

        #print in "hostname-interface ipaddr" format
        print "$hostname-$fields[0] $fields[1]\n";
         }
    }
}
}

Encrypted GRE Tunnel with ASA for Encryption Offload

2010-12-06T14:47:00.004-07:00

I have a requirement to connect two internal routers over a high-speed, 3rd party, non-Internet network. The two routers need to run EIGRP with each other. We want to encrypt the link, but the routers don't support IPSec in their current configuration. Purchasing IPSec acceleration hardware for them is expensive, but we happen to have two ASAs in inventory that aren't currently in production.

The simplest solution for this is to connect the two routers with a GRE tunnel, then use the ASAs to encrypt the GRE traffic.

While I've worked with ASAs quite a bit as stateful packet filters and as remote access VPN headends, it's been a really long time since I've used one in a point-to-point VPN. I thought I'd blog about the lab proof-of-concept so that I don't forget everything about the configuration.

My lab topology looks like this:

R4---ASA1---ASA2---R5

R4 and R5 have a standard GRE tunnel configured between them, running EIGRP to advertise their loopbacks. The tunnel destination is statically routed.

On R4:

interface Loopback0
ip address 4.4.4.4 255.255.255.0
!
interface FastEthernet0/0
description link to ASA1
ip address 1.1.1.4 255.255.255.0
!
interface Tunnel50
description GRE tunnel to R5, will be encrypted by ASA1
ip address 50.1.1.4 255.255.255.0
tunnel source FastEthernet0/0
tunnel destination 2.2.2.5
!
ip route 2.2.2.5 255.255.255.255 1.1.1.1
!
router eigrp 1
network 4.0.0.0
network 50.1.1.0 0.0.0.255

On R5:

interface Loopback0
ip address 5.5.5.5 255.255.255.0
!
interface FastEthernet0/0
description link to ASA2
ip address 2.2.2.5 255.255.255.0
!
interface Tunnel50
description GRE tunnel to R4, will be encrypted by ASA2
ip address 50.1.1.5 255.255.255.0
tunnel source FastEthernet0/0
tunnel destination 1.1.1.4
!
router eigrp 1
network 5.0.0.0
network 50.1.1.0 0.0.0.255
no auto-summary

[Note: the difference in the EIGRP configs isn't a mistake. The lab routers are running two different images, one of which has auto-summary disabled by default. The other has it enabled by default, so I had to explicitly turn it off.]

This is a pretty standard GRE configuration that is used all the time to make a virtual point-to-point circuit across any other network. I'm intentionally leaving out the MTU complications for now.

Here's the interface configuration for ASA1:

interface Ethernet0/2
description link to R4
nameif inside
security-level 100
ip address 1.1.1.1 255.255.255.0
!
interface Ethernet0/1
description link to ASA2 representing 3rd party network
nameif outside
security-level 0
ip address 100.1.1.1 255.255.255.0

and here's the same configuration from ASA2:

interface Ethernet0/2
description link to R5
nameif inside
security-level 100
ip address 2.2.2.1 255.255.255.0
!
interface Ethernet0/1
description link to ASA1 representing 3rd party network
nameif outside
security-level 0
ip address 100.1.1.2 255.255.255.0

Next, we need to configure IPSec on the ASAs. This is pretty similar to doing the same thing on a IOS router, with a couple of differences:
crypto ipsec transform-set ESP_3DES esp-3des esp-sha-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto map P2P_CRYPTO_CM 10 match address R5_PHYSICAL
crypto map P2P_CRYPTO_CM 10 set peer 100.1.1.1
crypto map P2P_CRYPTO_CM 10 set transform-set ESP_3DES
crypto map P2P_CRYPTO_CM interface outside
crypto isakmp enable outside
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
!
access-list R5_PHYSICAL extended permit ip host 2.2.2.5 host 1.1.1.4

When I first set this up, I forgot the unfamiliar crypto isakmp enable outside command, and it took me a few minutes of looking at debugs to figure out why the ASA was dropping the IKE Phase 1 packets.

The other part that's different from the IOS configuration is the presence of a "tunnel-group" that defines the tunnel type and the IKE pre-shared key:

tunnel-group 100.1.1.1 type ipsec-l2l
tunnel-group 100.1.1.1 ipsec-attributes
pre-shared-key *****

I also needed to turn off NAT control so that the ASA wouldn't drop packets without a pre-defined NAT translation:

no nat-control

The configuration on the other ASA is identical, except that the crypto ACL and peer addresses are reversed, just like they would be in an IOS configuration.

Detecting a Transparent Proxy with Wireshark/Tshark

2010-12-02T14:07:00.004-07:00

Recently I got pulled into a debate between two colleagues who were troubleshooting a problem where some users could access a website over SSL, and others couldn't. One person was arguing that the problem was caused by client misconfiguration, and the other was arguing that it wasn't. Following my mantra "when in doubt, capture packets", we captured some traffic and had a look. I'm not going to go through the entire troubleshooting process; rather I'm going to focus on what was ultimately causing the problem. Here's the packet sequence, output from Tshark with some of the TCP details removed to make it fit:

921 13.795492 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [SYN]
926 13.837731 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [SYN, ACK]
927 13.837753 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [ACK]
928 13.838086 10.100.100.192 -> 184.86.133.186 SSL Client Hello
932 13.840253 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [RST]
943 13.879563 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [ACK]
944 13.879588 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [RST]
945 13.880052 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [RST]
946 13.881491 184.86.133.186 -> 10.100.100.192 TLSv1 Server Hello
947 13.881513 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [RST]
948 13.881535 184.86.133.186 -> 10.100.100.192 TLSv1 Certificate, Server Hello Done
949 13.881545 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [RST]
950 13.881554 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [RST]
953 13.882063 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [RST]

The key thing to note here is the delta between the SYN and SYN/ACK: about 42ms. When I was viewing this in Wireshark I set a packet time reference on packet #921, using the "Ctrl T" keyboard shortcut; this makes it easier to see the delta values.

I then set another time reference on packet 931, the TLSv1 Client Hello. Immediately following this, less than 1 millisecond later, we see a RST come back from the server. Red flag! Since we already established the probable latency between the hosts as ~42ms using the SYN - SYN/ACK pair, this is extremely suspicious.

A few packets later, we see a TLSv1 Server Hello message inbound, AFTER the RST. The delta? Approximately 42ms, exactly what we'd expect.

I immediately inferred from this that a transparent proxy content filter was spoofing the RST due to something that it deemed to be objectionable content, and for whatever reason it wasn't notifying the user.

I talked to the administrator for the content filter, and indeed, a policy had been incorrectly applied to some users that blocked the content in question.

Another win for packet capture.

playing with VLAN-based QoS

2010-11-18T07:58:00.004-07:00

A friend of mine brought up an interesting question yesterday:

Will the "mls qos vlan-based" command mark packets if the packets aren't routed by the SVI where the service policy is applied?

I had never even considered this before, so I guessed "no" before reading the docs. Then I set it up in the lab, and the answer (superficially) still seemed to be "no". Then, however, I executed the old RTFM maneuver: the docs made it sound like the answer should be "yes". So I thought I'd have a closer look. Here are the details.

The test topology looks like this:

R5---Switch1---Switch2---Cat3750---R6

R5 & R6 are 2800 series routers.
Switch1 and Switch2 are 3560s.
Cat3750, unsurprisingly, is a Catalyst 3750, and the subject of the test.

All of the inter-switch links are 802.1q trunks, and the two routers are connected to access ports in VLAN 6.

R5's interface IP address is 10.6.6.5, with a loopback of 5.5.5.5.
R6's interface IP address is 10.6.6.6, with a loopback of 6.6.6.6.
R5 and R6 are running OSPF on all interfaces and are neighbors when the test begins.

First, I set up a QoS policy on the 3750 that would make packets distinguishable from each other:

ip access-list extended ALL
permit ip any any
!
class-map match-any FOO
match access-group name ALL
!
policy-map IN-MARKING
class FOO
set ip dscp cs2
class class-default
set dscp cs1

Then I applied the policy-map to an unrouted SVI for VLAN 6:

interface Vlan6
no ip address
service-policy input IN-MARKING

Then I set R6's access port for VLAN-based QoS:

interface FastEthernet1/0/2
switchport access vlan 6
switchport mode access
mls qos vlan-based
spanning-tree portfast

Since all traffic should be marked as CS2, the OSPF hello packets being exchanged between R5 and R6 should get marked immediately if the policy is working, but I also generated some ICMP and telnet traffic just for good measure.

My superficial, pre-RTFM diagnosis, said it wasn't working:

c3750#sh policy-map interface
Vlan6

Service-policy input: IN-MARKING

Class-map: FOO (match-any)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: access-group name ALL
0 packets, 0 bytes
5 minute rate 0 bps

Class-map: class-default (match-any)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: any
0 packets, 0 bytes
5 minute rate 0 bps

After my RTFM episode, I started wondering if the packets were getting marked anyway, but not counted by the switch. Idecided to try applying a service policy on R5 that would have rather dramatic effects if the packets were in fact marked:

class-map match-any CM_CS2
match ip dscp cs2
!
policy-map PM_DROP_CS2
class CM_CS2
drop

then...

R5(config)#int f0/0
R5(config-if)#service-policy input PM_DROP_CS2
R5(config-if)#
R5(config-if)#
*Nov 18 15:37:18.097: %OSPF-5-ADJCHG: Process 1, Nbr 20.1.1.6 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Dead timer expired

R5#sh policy-map interface f0/0 input class CM_CS2
FastEthernet0/0

Service-policy input: PM_DROP_CS2

Class-map: CM_CS2 (match-any)
20 packets, 1880 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: ip dscp cs2 (16)
20 packets, 1880 bytes
5 minute rate 0 bps
drop

So... in conclusion: the "mls qos vlan-based" command does indeed work for service policies applied to interfaces that don't route packets for the VLAN... essentially, the command causes the access port to inherit the QoS policy applied to the SVI, and the SVI doesn't count them. This does actually make sense, but it took working through the lab to make my brain process it correctly.

Links to Network World Blog Posts

2010-10-28T09:53:00.002-06:00

Last spring I wrote a series of guest blog posts on Network World, so I thought I'd link them here for future reference:

IOS Output Filters
More IOS Output Filters
Finding IP Addresses
NetFlow Part 1
NetFlow Part 2
NetFlow Part 3
When in Doubt, Capture Packets

Counting Regex Matches on Catalyst Switches

2010-09-27T21:32:00.002-06:00

As an old-time Unix/Linux guy, I’ve been wanting the IOS equivalent of the “wc -l” command for oh, about a million years now. Juniper has had this in JunOS for quite some time, via their “count” command. In Unix/Linux shell environments, “wc -l” counts the number of lines in the input. This is very useful for things like counting the number of unused ports on a switch. Until recently, I was stuck with using SSH one-liners to pipe IOS commands through a shell filter, like this:

$ ssh 10.1.1.2 'sh int status | i not|disa' | wc -l
Password:
19

In English: this executes the command “show interface status | include not|disa” via SSH on the remote device, which returns all the lines that are in either the “notconnected” or “disabled” status. Then it feeds those lines to the “wc -l” command, which counts the lines.

A couple of months ago, however, I noticed that recent versions of IOS for the Catalyst access switches (3560/3750, and probably the 2960--I haven’t tested the latter) have a “count” filter:

Switch#sh int status | count ?
LINE Regular Expression

Switch#sh int status | count not|disa
Number of lines which match regexp = 19

Anyway, nothing particularly groundbreaking here, just a nice feature that’s long overdue--I hope it gets ported into other IOS images as well.

Cisco Live 2010

2010-07-02T22:01:00.004-06:00

This was my 6th consecutive year at CiscoLive (formerly known as Networkers), and as always I had a great time. I thought I'd write down a few brief thoughts on my sessions and other experiences this year.

802.1x 8-Hour Techtorial
This was the first time I've done a "techtorial" (Cisco's term for a 4 or 8 hour expanded seminar that costs extra) since 2006, and it was probably the best one I've done. All of the presenters were excellent, and they did a great job of keeping the class engaged by switching frequently between lecture and live demonstrations by three different instructors. They also included a real-world case study, presented by the actual customer involved (a large Canadian university). Cisco has a tendency to make up artificial case studies (or anonymize them to the point of making them pointless), so it was great to see a live customer on stage, presenting the entire implementation process, warts and all.

LISP - A Next Generation Networking Architecture
I have read a fair bit about LISP over the last couple of years, but this was the first time I've gone to a session on it. Basically, Dino and company are trying to solve three or four of the biggest problems in networking in one fell swoop: 1) global routing table size, 2) certain types of IPv4/IPv6 transport issues, 3) virtual machine mobility, and possibly 4) other mobility problems. I really don't have the background to evaluate a protocol that's designed to solve extremely difficult problems at a global scale, but it was fascinating to see the thought process and design issues involved.

Routed Fast Convergence and High Availability
I have been hearing about this session for years, and it didn't disappoint. I was already familiar with most of the tools discussed, but the devil is in the details, and I came away with a much better understanding of many techniques used to achieve fast convergence in routed networks.

Smart Grid: Developing a Communications Architecture for the Utility of the Future
I didn't intend to visit this session initially, but the speaker was late for my scheduled session, and I had no interest in sitting around waiting for him to show up. This session was happening nearby, and I knew that Bill Parkhurst is something of a network architecture guru, so it was an easy pick. Even though I have absolutely no background in the electrical utility world, this was a very useful session from a general professional development perspective.

Unified HA Network Design: The Evolution of the Next Generation Network
If I could recommend only one session on large-scale network design, this would be it. These guys are working on the largest, most failure-sensitive networks on the planet, and they're giving away what they've learned. What more needs to be said?

Advanced Security Management & Incident Response
This was my second time attending this session (the last time was in 2007), and it was definitely worth attending a second time. I really like operationally-focused sessions (as opposed to product-focused ones), and that's what this one is all about. The presenters are front-line senior incident responders in Cisco's internal security organization, and it's great to see how they deploy Cisco tools and even (gasp) non-Cisco tools to respond to actual security incidents. I really hope they can convince the powers-that-be to let them run this as an expanded, 8-hour techtorial on security operations.

Those are the highlights of my sessions. I didn't attend a single session this year that was actually bad, but those were the ones that stood out the most.

Other thoughts: the meals were above average compared to previous years, except for breakfast. I hate getting to a breakfast and seeing nothing but bread products and some random, lonely looking fruit. The CCIE party was great; probably the best one I've attended. The CCIE NetVet reception with John Chambers was also excellent. My biggest complaint continues to be the ominous warnings about a $100 fee to replace a lost conference badge. I've never lost mine, but this just seems patently ridiculous.

As always, though, simply meeting other networking-focused professionals and renewing friendships with people I've known from previous years was the best part of the show.

Hoping to be back next year!

guest blogging

2010-05-05T19:30:00.001-06:00

I'm guest blogging on IOS tips and tricks this month at Network World.

Dan Geer on Cybersecurity

2010-04-16T15:16:00.002-06:00

I love Dan Geer's article "Cybersecurity and National Policy".

Wireshark Network Analysis

2010-04-09T10:49:00.004-06:00

For the last week I've been reading Laura Chappell's new book, Wireshark Network Analysis. I pre-ordered the book and was looking forward to it eagerly.

Overall, it's a superb book. First and foremost, I appreciate the fact that the writing has some personality to it. I've really enjoyed the author's footnotes, anecdotes, and humor.

Second, I like the fact that it's exhaustively thorough with both features and examples. There are relatively few cases where the Chappell simply describes a feature without offering an example of how it might be used in practice. Where she does so, the example is obvious enough to be unnecessary.

Particularly good are the examples of practical Wireshark tips and tricks in the protocol-specific sections. As an advanced Wireshark user with a thorough background in the internals of common network protocol operations, I was worried that this would be just another set of explanations of how various protocols work. Those explanations are there, but they are interspersed with many great tips and tricks about how to analyze the protocols more quickly and efficiently with Wireshark. I kept thinking, "why didn't I ever think of using that trick before?"

There is a lot here for both beginning and advanced Wireshark users.

I don't have a lot of criticisms to make: I thought the security sections were interesting, but a little dated. Exotic network and transport layer attacks just aren't all that common anymore; it would have been cool to have seen some analysis of a modern application-layer attack in action.

The other thing I would have liked is a discussion of the Lua scripting-language extensions to Wireshark. There is very little out there on the Internet about this so far, and most of what exists is oriented toward expert-level programmers. I was a bit disappointed to not even find Lua in the index. Still, the book is already huge, and adding a section on scripting might have made it unreasonably long. Maybe a volume 2?

Summary: if you use Wireshark, just buy it. This is an amazingly practical, hands-on book that will make you faster and more productive when analyzing network captures.

Blogging disclaimer: I paid full retail price for the book (and managed to somehow miss all the promo coupons) and am not being compensated in any way for this review.

computer crime and the small/medium business

2010-01-04T17:57:00.002-07:00

Good post on the little-known computer crime wave targeting small/medium businesses:

http://www.krebsonsecurity.com/2010/01/buried-warning-signs-2/

predicting reload times on Catalyst 3560/3750

2009-12-31T08:42:00.005-07:00

During a recent IOS upgrade on a Catalyst 3560, I was connected to the console and noticed that the reload was taking much longer than usual due to some operations by the "Front End Microcode IMG Mgr". The output looked like this:

POST: PortASIC RingLoopback Tests : Begin
POST: PortASIC RingLoopback Tests : End, Status Passed

front_end/ (directory)
extracting front_end/fe_type_1 (34760 bytes)
extracting front_end/front_end_ucode_info (86 bytes)
extracting front_end/fe_type_2 (73104 bytes)
extracting ucode_info (76 bytes)

Front-end Microcode IMG MGR: Installed 3 image(s) in cache:

Front-end Microcode IMG MGR: found microcode images for 3 devices.
Image for front-end 0: flash:/front_end_ucode_cache/ucode.1
Image for front-end 7: flash:/front_end_ucode_cache/ucode.1
Image for front-end 14: flash:/front_end_ucode_cache/ucode.1

Front-end Microcode IMG MGR: Preparing to program device microcode...
Front-end Microcode IMG MGR: Preparing to program device[0]...26580 bytes.
Front-end Microcode IMG MGR: Programming device 0...rwRrrrrrrwsssspsssspsssspsss
spsssspsssspsssspsssspsssspsssspsssspsssspsssspsssspsssspsssspsssspsssspsssspsss

[output truncated]

I opened a TAC case to find out what this is, since if you are relying on highly predictable reload times during a maintenance window, this could throw a wrench into your plans.

It turns out that the Catalyst switches have a special-purpose microcontroller that rarely needs to be upgraded. When it does need upgrading, however, the upgrade happens as a normal part of a new IOS image load. This upgrade makes the first reload to the new IOS take much longer than usual--I didn't time it, but I would guess 3-4 times longer than normal.

Microcontroller upgrades are not typically listed in the image release notes, so the only way to know for sure how long a particular upgrade is going to take is to test it in a lab, using the exact same before/after images that you will use in production.

ACLs and TCAMs in Catalyst Switches

2009-12-21T13:19:00.004-07:00

One of the things you need to look at when designing networks with Catalyst switches is the potential for TCAM exhaustion due to ACL and QoS configuration. Here are a couple of documents that explain the issue:

Catalyst 6500
http://www.cisco.com/en/US/products/hw/switches/ps708/products_white_paper09186a00800c9470.shtml

Catalyst 4500 and 4900 Series
http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a008054a499.shtml

ten steps of small LAN design

2009-12-15T10:25:00.004-07:00

A few days ago I posted an amusing comment on Ivan Pepelnjak's always excellent Cisco IOS Hints and Tricks blog, and he found it funny enough to create a separate post. I'll repeat my ten step program here for future reference:

Build everything at layer 2 because "it's simpler".
Scale a little.
Things start breaking mysteriously. Run around in circles. Learn about packet sniffers and STP.
Learn about layer 3 features in switches you already own. Start routing.
Scale more.
Things start breaking mysteriously. Learn about TCAMs. Start wishing for NetFlow.
Redesign. Buy stuff.
Scale more.
VMWare jockeys start asking about bridging across the WAN.
Enroll in hair loss program.

incoming dial-peers

2009-12-15T10:11:00.004-07:00

I had an interesting troubleshooting experience that showed me that I didn't fully understand how incoming dial-peers work with POTS lines.

I had a simple H.323 config that hands off a call arriving on an FXO port to CallManager:

voice-port 1/0/2
connection plar 7001
description POTS line
caller-id enable

dial-peer voice 7001 voip
destination-pattern 700.
session target ipv4:10.1.1.100
dtmf-relay h245-alphanumeric

When a call was placed to the line connected to the FXO port on 1/0/2, the call would be sent to the IP phone with the wrong caller ID.

I ran a "debug voip ccapi" and discovered that the incoming dial-peer was not the default dial-peer 0, but another dial-peer (numbers sanitized):

dial-peer voice 1000 pots
description 555-1212
destination-pattern 1212
clid network-number 9705551212
port 1/0/2

This dial-peer had accidentally been left active from a prior configuration, and its "clid network-number" command was thus overwriting the correct caller ID.

I didn't know this previously, but it turns out that an incoming POTS dial peer is matched if it has a "port" statement equal to the inbound voice-port, AND any one of the following three commands is present:

incoming called-number
answer-address
destination-pattern

Removing the destination-pattern command or removing the dial-peer entirely corrects the problem and causes dial-peer 0 to be matched inbound.

simple exclusion filters

2009-12-08T17:35:00.002-07:00

I use these constantly (and many others, but these come first to mind):

display only interfaces with assigned IP addresses:
sh ip int b | e una

display only active switch interfaces:
sh int status | e not

display CDP neighbors, except phones:
sh cdp n | e SEP

RIP database and administrative distance

2009-06-23T10:19:00.002-06:00

I was helping a friend study for CCNA the other day and saw a RIP behavior I'd never noticed before. I knew that RIP keeps a local route database that is displayed with the show ip rip database command. If another route to the same prefix with a better administrative distance is preferred in the global routing table, however, the RIP database doesn't show the route. This is different than more sophisticated routing protocols in which a prefix is kept in the protocol-specific topology table even if a route from another protocol with a better AD is in the global routing table.

Cisco IPS Manager Express

2009-06-10T07:05:00.005-06:00

I've been doing Cisco IDS/IPS stuff recently for the first time in a long while. If you haven't tried Cisco's new free IPS Manager Express application, check it out. It makes IDS/IPS event monitoring and management reasonably useful and almost pain-free. The interface is much more intuitive that other Cisco IDS/IPS GUI products. The only problem is that the current version supports only 5 sensors; supposedly this will increase in a future release.

Added 12/15/09:
The latest version of IME supports 10 sensors.

open ports on IOS router

2009-06-10T06:59:00.002-06:00

Haven't posted here in ages. Interesting trivia: the old "show ip sockets" command doesn't work in new 12.4T images. It's been replaced by "show control-plane host open-ports":

#sh control-plane host open-ports
Active internet connections (servers and established)
Prot Local Address Foreign Address Service State
tcp *:22 *:0 SSH-Server LISTEN
tcp *:23 *:0 Telnet LISTEN
tcp *:15904 x.x.x.x:179 IOS host service ESTABLIS
tcp *:179 x.x.x.x:38441 BGP ESTABLIS
tcp *:179 *:0 BGP LISTEN
tcp *:179 *:0 BGP LISTEN
tcp *:179 *:0 BGP LISTEN
udp *:49 x.x.x.x:0 TACACS service LISTEN
udp *:161 *:0 IP SNMP LISTEN
udp *:162 *:0 IP SNMP LISTEN
udp *:57421 *:0 IP SNMP LISTEN
udp *:1985 *:0 cisco HSRP LISTEN

how to clone a VM in the free VMWare ESXi

2009-03-02T22:42:00.000-07:00

In the free verison of ESXi, it's not obvious how to clone a VM, since you don't have VirtualCenter available.

I've read about some people using the free version of Converter, but this didn't work for me (Converter keeps hanging partway through the operation). Here's what I did; note that I'm running local storage only on an older host machine:

1) From the ESXi console, hit Alt-F1, then type "unsupported". You will get a bunch of dire warnings about this being an unsupported mode. You are now in a bare-bones Unix shell.

2) (optional) Enable ssh so you can do the rest remotely: use vi to edit the /etc/inetd.conf file and uncomment the line that starts with "ssh". Exit and restart inetd with "kill -HUP " where is the process ID of inetd. You can find the PID with "ps aux | grep inetd".

3) cd /vmfs/volumes/datastore

4) Use vmkfstools to clone the .vmdk file:

# vmkfstools -i imageA/imageA.vmdk imageB/imageB.vmdk
Destination disk format: VMFS thick
Cloning disk 'xubu1/xubu1.vmdk'...
Clone: 100% done.

5) From VI Client, create a new VM and select the custom option. When you get to the "select a hard disk" part, select the VMDK file you just cloned in the previous step.

6) You may have trouble with a cloned machine in Windows; you'll need to run sysprep to make it unique. In Linux you can just change the IP (if not using DHCP) and edit /etc/hostname and reboot to make a unique hostname.

steps to install Bro IDS on Ubuntu

2009-02-28T21:19:00.000-07:00

This works on both Ubuntu and Xubuntu. Use sudo on everything or run as root.

apt-get install libncurses5-dev g++ bison flex
apt-get install libmagic-dev libgeoip-dev libpcap-dev libssl-dev
tar -xvf bro-1.4-release.tar.gz
cd bro-1.4
./configure --prefix=/usr/local/bro
make
make install

How to Install dig for Windows

2009-02-26T13:29:00.000-07:00

dig is the standard tool for advanced DNS queries. A Windows version is available as part of the BIND port. To install it on Windows:

1) Go to ftp://ftp.isc.org/isc/bind9/9.5.0-P2/

2) Download BIND9.5.0-P2.zip

3) Open the archive with WinZip

4) Extract dig.exe and *.dll to c:\windows\system32

5) If you want the documentation page, extract dig.html to somewhere that you can find it.

Now you will be able to use dig from your command prompt in Windows. It is faster and more sophisticated than nslookup.

Get the quick help options with "dig -h".

port-map feature

2008-11-19T16:00:00.000-07:00

Ever forget what port number maps to what service? A router running Adv IP Services, Adv Security, or Adv Enterprise Services will tell you all the common ones using the show ip port-map command, which is part of the IOS firewall feature set:

Router#sh ip port-map
Default mapping: snmp udp port 161 system defined
Default mapping: echo tcp port 7 system defined
Default mapping: echo udp port 7 system defined
Default mapping: telnet tcp port 23 system defined
Default mapping: wins tcp port 1512 system defined
Default mapping: n2h2server tcp port 9285 system defined
Default mapping: n2h2server udp port 9285 system defined
Default mapping: nntp tcp port 119 system defined
Default mapping: pptp tcp port 1723 system defined
Default mapping: rtsp tcp port 554,8554 system defined
Default mapping: bootpc udp port 68 system defined
Default mapping: gdoi udp port 848 system defined
Default mapping: tacacs udp port 49 system defined

[output truncated]

You can, of course, filter for stuff you find interesting:

Router#sh ip port-map | i 110
Default mapping: pop3 tcp port 110 system defined

in-line editing of Cisco ACLs

2008-11-13T11:05:00.000-07:00

Many people don't realize that reasonably recent versions of IOS have in-line ACL editing.

The way this works is that ACLs have invisible line numbers that only show up when using a show access-list command, not when doing "show run":

R1#sh access-list TEST
Extended IP access list TEST
10 permit tcp host 1.1.1.1 host 2.2.2.2
20 permit gre host 3.3.3.3 host 4.4.4.4

In the example above, the line numbers are shown. If we just look at the config, they're not:

R1#sh run | s access-list
ip access-list extended TEST
permit tcp host 1.1.1.1 host 2.2.2.2
permit gre host 3.3.3.3 host 4.4.4.4

Now, let's say I want to make line number 10 more restrictive:

R1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)#ip access-list ext TEST
R1(config-ext-nacl)#15 permit tcp host 1.1.1.1 host 2.2.2.2 eq www
R1(config-ext-nacl)#do sh access-list TEST
Extended IP access list TEST
10 permit tcp host 1.1.1.1 host 2.2.2.2
15 permit tcp host 1.1.1.1 host 2.2.2.2 eq www
20 permit gre host 3.3.3.3 host 4.4.4.4
R1(config-ext-nacl)#no 10
R1(config-ext-nacl)#do sh access-list TEST
Extended IP access list TEST
15 permit tcp host 1.1.1.1 host 2.2.2.2 eq www
20 permit gre host 3.3.3.3 host 4.4.4.4

If the odd sequence numbers really bother you, you can fix them:

R1(config)#ip access-list resequence TEST ?
<1-2147483647> Starting Sequence Number

R1(config)#ip access-list resequence TEST 10 ?
<1-2147483647> Step to increment the sequence number

R1(config)#ip access-list resequence TEST 10 10 ?

R1(config)#ip access-list resequence TEST 10 10
R1(config)#do sh access-list TEST
Extended IP access list TEST
10 permit tcp host 1.1.1.1 host 2.2.2.2 eq www
20 permit gre host 3.3.3.3 host 4.4.4.4

I'm not sure what IOS version this appeared in, but it's been around since at least the early 12.3T images.

GRE tunnels and OSPF adjacencies

2008-10-29T09:36:00.000-06:00

I made the mistake of starting to play with one of Ivan Pepelnjak's OSPF challenges on his great IOS blog, and got carried away playing with GRE tunnels instead.

On R1:

interface Loopback0
ip address 10.1.2.1 255.255.255.0

interface Serial1/0
ip unnumbered Loopback0
encapsulation ppp

interface Tunnel0
ip unnumbered Loopback0
ip ospf 1 area 1
tunnel source Serial1/0
tunnel destination 10.1.2.3

On R2:

interface Loopback0
ip address 2.2.2.2 255.255.255.0

interface Serial1/0
ip address 10.1.2.3 255.255.255.248
encapsulation ppp

interface Tunnel0
ip address 22.22.22.22 255.255.255.0
ip ospf 1 area 1
tunnel source Serial1/0
tunnel destination 10.1.2.1

This configuration forms no adjacency because the GRE tunnel interfaces are on different networks. A debug confirms this:

R2#deb ip os adj
OSPF adjacency events debugging is on
R2#
*Oct 29 09:43:03.123: OSPF: Rcv pkt from 10.1.2.1, Tunnel0, area 0.0.0.1 : src not on the same network

from a debug ip packet at the same time:

*Oct 29 10:08:11.015: IP: s=22.22.22.22 (Tunnel0), d=224.0.0.5, len 76, rcvd 0

[incidentally, I'm not sure why this behavior doesn't violate section 8.2 of RFC 2328, which states:

  ...the packet's IP source address is required to
be on the same network as the receiving interface.  This
can be verified by comparing the packet's IP source
address to the interface's IP address, after masking
both addresses with the interface mask.  This comparison
should not be performed on point-to-point networks. On
point-to-point networks, the interface addresses of each
end of the link are assigned independently, if they are
assigned at all.

GRE tunnels are considered point-to-point networks by OSPF.]

However, if I remove the explicitly configured IP address on R2's tunnel interface and replace it with an unnumbered configuration, using a loopback that's still not in the same network, it works:

R2(config)#int tu 0
R2(config-if)#ip unn lo 0
R2(config-if)#^Z
R2#
*Oct 29 09:44:44.907: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.1 on Tunnel0 from LOADING to FULL, Loading Done

R2#sh ip os n

Neighbor ID Pri State Dead Time Address Interface
10.0.0.1 0 FULL/ - 00:00:33 10.1.2.1 Tunnel0

R1#sh ip os n

Neighbor ID Pri State Dead Time Address Interface
10.1.2.3 0 FULL/ - 00:00:32 2.2.2.2 Tunnel0

At first I thought that maybe the router was changing the source address for OSPF packets to the address of the tunnel source, but that's not the case:

R1#deb ip packet
IP packet debugging is on
R1#
*Oct 29 10:05:44.955: IP: s=10.1.2.1 (local), d=224.0.0.5 (Tunnel0), len 80, sending broad/multicast
*Oct 29 10:05:44.955: IP: s=10.1.2.1 (Tunnel0), d=10.1.2.3 (Serial1/0), len 104, sending
*Oct 29 10:05:45.703: IP: s=2.2.2.2 (Tunnel0), d=224.0.0.5, len 80, rcvd 0

As you can see, the OSPF hellos are coming from the loopback IP of R2, which is on a completely different network.

IP phones show bogus "unknown" status in UCM

2008-10-17T11:58:00.001-06:00

If you end up with a bunch of normally functioning, non-problematic IP phones showing up as "unknown" instead of "registered" in CallManager administration, you can restart the RIS Data Collector service on the publisher to solve the problem. This doesn't affect call processing, but it does seem to restart some media resources like the annunciator, etc.

certification musings

2008-10-13T14:43:00.001-06:00

I started this blog intending it as a repository for odd network and VoIP related issues I run into over time, but I just have to vent a bit about a recent experience.

Last week I interviewed someone whose resume listed virtually every non-CCIE Cisco certification there is: CCNA, CCNP, CCSP, CCVP, CCDP, etc etc.

Now, I'm not naive enough to think that someone with all those certifications will necessarily be thoroughly proficient at all the subject areas (I have the CCSP among other things, but I'm pretty rusty on Cisco IDS), but a passing familiarity would be nice. This individual couldn't describe basic ARP operation, a basic TCP handshake, and couldn't name any layer 2 WAN protocols. He didn't do any better on voice or security questions.

Obviously it's possible to get all these certifications without knowing much, but if you're going to put them on your resume, at least try to be familiar with the basic concepts in the subject area.

Better yet, use the tests as a way to evaluate your REAL theoretical and practical knowledge of the subject area, not as something to spiff up your resume.

update on yesterday's application weirdness post

2008-10-09T08:58:00.000-06:00

A friend read my post yesterday on odd problems with H.323 call setup on VT-Advantage-enabled phones after changing media resources configuration. He observed that IOS 12.4(20T), which we're running on the ISRs containing the DSPs, is the first IOS image to fully implement H.320 ISDN video conferencing capability, and that this might have something to do with the problem. The H.323 gateway that's actually handing the call to the PSTN isn't running 12.4(20T); it's running a much older image. Why the code on the DSP farm would affect a call on a different gateway, I don't understand, but it certainly seems like too much correlation to be coincidence.

Weird application interactions

2008-10-08T16:15:00.001-06:00

Recently we made some changes to our media resource configuration in CallManager 6.x. Specifically, we stopped using DSPs on a 6608 blade in a Catalyst 6500 and started using DSPs in a couple of 2811 routers. Immediately after this change, a couple of users reported that they could no longer make outbound PSTN calls that transit a particular H.323 gateway. The weirdest part was that if they rebooted their 7940 phone, they could make a single call, but all subsequent calls would fail with a fast busy until they rebooted again. Calls through other gateways worked fine.

A ISDN Q.931 debug showed that the second outbound call was setting up as "unrestricted digital" instead of "speech".

I went 'round with TAC about this for a few emails and they were getting ready to blame the telco (?!) when I realized that the thing these two users have in common is VT Advantage (a Cisco desktop video-conferencing application). I asked them to turn off VT Advantage, and presto, all calls now worked.

Upgrading to the latest VT Advantage version solved the problem completely.

Why a change in media resources would affect outbound calling through a single H.323 gateway on users with an old version of VT Advantage is beyond me, but that's my story and I'm sticking to it.

Unity and Multiple Active Directory Domains

2008-10-08T08:42:00.000-06:00

There seems to be a lack of information out there, even with some TAC engineers, about how to prepare Cisco Unity to import subscribers from Exchange servers in AD domains different from the one in which Unity is installed. I've even had several Cisco engineers tell me that this is unsupported, which is totally false. As far as I know, these are the requirements for Unity with respect to multiple domains:

1) All Exchange servers from which you want to import subscribers must be in the same AD forest. Unity unified messaging does not work across multiple forests. Of course, you can still create voicemail boxes that are totally independent of AD, but that's not the point of unified messaging. Exchange servers in different domains in the same forest is not a problem, as long as the PES (see below) is in the same domain as Unity.

2) The Unity server(s) must be installed in the same Active Directory domain as the partner Exchange server, or PES. You specify the PES during the Unity install process. It can be changed with the Message Store Integration Wizard. The PES must have Exchange routing group connectors to all other Exchange servers from which you want to import subscribers. More info on changing the PES is here.

3) You must run the Permissions Wizard on the Unity server while logged in as an account that can set permissions in all the domains containing Exchange servers from which you want to import subscribers. This might require running the PW multiple times under different accounts. This works fine. You may need to add a domain admin account from another domain to the local admin group on the Unity server. When you run PW from accounts in different domains, it will probably fail on some of the domains. This is because you probably won't have a single account that has all the required permissions in all the domains. You just need to continue running PW under different accounts until it has succeeded in all the domains that have Exchange servers from which you're importing subscribers.

Changes don't take effect in UCM 6

2008-09-16T09:36:00.001-06:00

If you make a change to a Cisco Unified Communications Manager 6.x system and it doesn't seem to take effect, try restarting the Database Layer Monitor service and the TFTP service. This can be done during production hours.

In my case, the change in question was a change to the CallManager Group setting in a device pool.

Unity Permissions Wizard Stuff to Remember

2008-05-31T15:47:00.000-06:00

To successfully run Unity Permission Wizard in a new domain, you need to run it as the Domain Admin in the remote domain. It will fail on the existing domains (the permissions are already set correctly on them, assuming things are already working), and succeed in the new domain.

Trivia Friday: ip subnet-zero

2008-04-18T13:02:00.000-06:00

The ip subnet-zero command has been largely irrelevant since it became the default in IOS quite some time ago, but it still generates occasional discussion among consumers of IOS trivia.

For whatever reason, a lot of people seem to think that setting no ip subnet-zero causes you to be unable to use either the all-zeros OR all-ones subnets of a classful network. This is not the case. Turning off ip subnet-zero (not that you would want to do this outside of an IOS trivial pursuit game!) only prevents you from assigning the all-zeros subnet of a network:


R2(config)#no ip subnet-zero
R2(config)#int loopback 0
R2(config-if)#ip add 10.0.0.1 255.255.255.192
Bad mask /26 for address 10.0.0.1
R2(config-if)#ip add 10.255.255.254 255.255.255.192
R2(config-if)#ip add 192.168.0.1 255.255.255.192
Bad mask /26 for address 192.168.0.1
R2(config-if)#ip add 192.168.254.254 255.255.255.192
R2(config-if)#

Note that we are only prevented from assigning the all-zeros subnet of the network, and there's no restriction on assigning the all-ones subnet.

CallManager Trace Decoding Tools

2008-04-14T16:54:00.001-06:00

Triple Combo looks to be a nice tool for decoding CCM/UCM trace files:

http://www.employees.org/~tiryaki/tc/

In the past I've also used TranslatorX, which is good but has some missing pieces:

http://www.employees.org/~pgiralt/TranslatorX/

changing IP address on Unity Express

2008-04-01T09:17:00.000-06:00

Apparently you are supposed to shut down CUE before you change the IP address on the service module interface... I didn't, and the CUE module stopped accepting calls until it was reset with the "service module service-engine 0/1 reset" command.

troubleshooting basic POTS calls on a router

2008-03-14T15:44:00.001-06:00

We get calls fairly regularly from users at sites with router-based VoIP systems (whether UCM Express or distributed UCM, it doesn't matter) saying that callers to their outside lines are getting ring-no-answer behavior.

The first thing I want to do is see if the calls are actually getting to the router. So I access the CLI and do the following:

Router#terminal monitor
Router#debug vpm signal

Then I place a test call to one of their outside lines. If the router just sits there and produces no output, then the problem is almost certainly either in the PSTN, or somebody has unplugged the cable from the FXO port.

If the router produces a bunch of barely comprehensible output, then the line is ringing and you've got further troubleshooting to do. Don't forget to turn off debugging when you're done!

DSCP Confusion

2008-03-14T15:15:00.001-06:00

A friend of mine asked me a question the other day that pointed out a common area of confusion with the DiffServ model for IP QoS. He was reading some documentation that recommended setting the DSCP value for voice-signaling packets to 26 instead of 24. He was trying to convert those numbers to Assured Forwarding (AF) values, and came up with 31 and 30, respectively. His question was "Why does the documentation appear to be making the QoS treatment of these packets WORSE?"

The answer is, it's not. 30 is not a valid AF value. With the Assured Forwarding classes, there are three drop precedence values: 1, 2, and 3. Since zero is not a valid drop precedence, 30 is not a valid AF.

The DSCP value of 26 is 001000 in binary. The per-hop-behavior (PHB) value for this is actually not an Assured Forwarding PHB at all, it's a Class Selector value, CS3. The class selector values equate directly to the old IP precedence values, from the days before DiffServ. So DSCP 26 = CS3 = IP precedence 3 = 001000.

Clear as mud? Try the Wikipedia article.

IPv6 at IETF

2008-03-14T14:19:00.000-06:00

A few weeks ago at NANOG they turned off IPv4 on the wireless network for a while. Here's a story about the same thing at the much larger IETF meeting.

MBXSuite tool for troubleshooting Unity mailbox issues

2008-03-14T14:06:00.000-06:00

I recently learned about a useful tool for troubleshooting Unity/Exchange mailbox issues. It's located in the \CommServer\TechTools folder, and it's called MBXSuite.exe.

To use it:

1) Select the user whose mailbox you want to test with.
2) Select the Unity service account you want to run as--usually UnityMsgStoreSvc.
3) Check "Verbose Diags"
4) Click "Logon Mailbox" and check the Diagnostics pane for errors.