predicting reload times on Catalyst 3560/3750

During a recent IOS upgrade on a Catalyst 3560, I was connected to the console and noticed that the reload was taking much longer than usual due to some operations by the "Front End Microcode IMG Mgr". The output looked like this:

POST: PortASIC RingLoopback Tests : Begin
POST: PortASIC RingLoopback Tests : End, Status Passed

front_end/ (directory)
extracting front_end/fe_type_1 (34760 bytes)
extracting front_end/front_end_ucode_info (86 bytes)
extracting front_end/fe_type_2 (73104 bytes)
extracting ucode_info (76 bytes)

Front-end Microcode IMG MGR: Installed 3 image(s) in cache:

Front-end Microcode IMG MGR: found microcode images for 3 devices.
Image for front-end 0: flash:/front_end_ucode_cache/ucode.1
Image for front-end 7: flash:/front_end_ucode_cache/ucode.1
Image for front-end 14: flash:/front_end_ucode_cache/ucode.1

Front-end Microcode IMG MGR: Preparing to program device microcode...
Front-end Microcode IMG MGR: Preparing to program device[0]...26580 bytes.
Front-end Microcode IMG MGR: Programming device 0...rwRrrrrrrwsssspsssspsssspsss
spsssspsssspsssspsssspsssspsssspsssspsssspsssspsssspsssspsssspsssspsssspsssspsss

[output truncated]

I opened a TAC case to find out what this is, since if you are relying on highly predictable reload times during a maintenance window, this could throw a wrench into your plans.

It turns out that the Catalyst switches have a special-purpose microcontroller that rarely needs to be upgraded. When it does need upgrading, however, the upgrade happens as a normal part of a new IOS image load. This upgrade makes the first reload to the new IOS take much longer than usual--I didn't time it, but I would guess 3-4 times longer than normal.

Microcontroller upgrades are not typically listed in the image release notes, so the only way to know for sure how long a particular upgrade is going to take is to test it in a lab, using the exact same before/after images that you will use in production.

ACLs and TCAMs in Catalyst Switches

One of the things you need to look at when designing networks with Catalyst switches is the potential for TCAM exhaustion due to ACL and QoS configuration. Here are a couple of documents that explain the issue:

Catalyst 6500
http://www.cisco.com/en/US/products/hw/switches/ps708/products_white_paper09186a00800c9470.shtml

Catalyst 4500 and 4900 Series
http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a008054a499.shtml

ten steps of small LAN design

A few days ago I posted an amusing comment on Ivan Pepelnjak's always excellent Cisco IOS Hints and Tricks blog, and he found it funny enough to create a separate post. I'll repeat my ten step program here for future reference:

1. Build everything at layer 2 because "it's simpler".
2. Scale a little.
3. Things start breaking mysteriously. Run around in circles. Learn about packet sniffers and STP.
4. Learn about layer 3 features in switches you already own. Start routing.
5. Scale more.
6. Things start breaking mysteriously. Learn about TCAMs. Start wishing for NetFlow.
7. Redesign. Buy stuff.
8. Scale more.
9. VMWare jockeys start asking about bridging across the WAN.
10. Enroll in hair loss program.

incoming dial-peers

I had an interesting troubleshooting experience that showed me that I didn't fully understand how incoming dial-peers work with POTS lines.

I had a simple H.323 config that hands off a call arriving on an FXO port to CallManager:

voice-port 1/0/2
connection plar 7001
description POTS line
caller-id enable

dial-peer voice 7001 voip
destination-pattern 700.
session target ipv4:10.1.1.100
dtmf-relay h245-alphanumeric

When a call was placed to the line connected to the FXO port on 1/0/2, the call would be sent to the IP phone with the wrong caller ID.

I ran a "debug voip ccapi" and discovered that the incoming dial-peer was not the default dial-peer 0, but another dial-peer (numbers sanitized):

dial-peer voice 1000 pots
description 555-1212
destination-pattern 1212
clid network-number 9705551212
port 1/0/2

This dial-peer had accidentally been left active from a prior configuration, and its "clid network-number" command was thus overwriting the correct caller ID.

I didn't know this previously, but it turns out that an incoming POTS dial peer is matched if it has a "port" statement equal to the inbound voice-port, AND any one of the following three commands is present:

incoming called-number
answer-address
destination-pattern

Removing the destination-pattern command or removing the dial-peer entirely corrects the problem and causes dial-peer 0 to be matched inbound.

simple exclusion filters

I use these constantly (and many others, but these come first to mind):

display only interfaces with assigned IP addresses:
sh ip int b | e una

display only active switch interfaces:
sh int status | e not

display CDP neighbors, except phones:
sh cdp n | e SEP

RIP database and administrative distance

I was helping a friend study for CCNA the other day and saw a RIP behavior I'd never noticed before. I knew that RIP keeps a local route database that is displayed with the show ip rip database command. If another route to the same prefix with a better administrative distance is preferred in the global routing table, however, the RIP database doesn't show the route. This is different than more sophisticated routing protocols in which a prefix is kept in the protocol-specific topology table even if a route from another protocol with a better AD is in the global routing table.

Cisco IPS Manager Express

I've been doing Cisco IDS/IPS stuff recently for the first time in a long while. If you haven't tried Cisco's new free IPS Manager Express application, check it out. It makes IDS/IPS event monitoring and management reasonably useful and almost pain-free. The interface is much more intuitive that other Cisco IDS/IPS GUI products. The only problem is that the current version supports only 5 sensors; supposedly this will increase in a future release.

Added 12/15/09:
The latest version of IME supports 10 sensors.

open ports on IOS router

Haven't posted here in ages. Interesting trivia: the old "show ip sockets" command doesn't work in new 12.4T images. It's been replaced by "show control-plane host open-ports":

#sh control-plane host open-ports
Active internet connections (servers and established)
Prot Local Address Foreign Address Service State
tcp *:22 *:0 SSH-Server LISTEN
tcp *:23 *:0 Telnet LISTEN
tcp *:15904 x.x.x.x:179 IOS host service ESTABLIS
tcp *:179 x.x.x.x:38441 BGP ESTABLIS
tcp *:179 *:0 BGP LISTEN
tcp *:179 *:0 BGP LISTEN
tcp *:179 *:0 BGP LISTEN
udp *:49 x.x.x.x:0 TACACS service LISTEN
udp *:161 *:0 IP SNMP LISTEN
udp *:162 *:0 IP SNMP LISTEN
udp *:57421 *:0 IP SNMP LISTEN
udp *:1985 *:0 cisco HSRP LISTEN

how to clone a VM in the free VMWare ESXi

In the free verison of ESXi, it's not obvious how to clone a VM, since you don't have VirtualCenter available.

I've read about some people using the free version of Converter, but this didn't work for me (Converter keeps hanging partway through the operation). Here's what I did; note that I'm running local storage only on an older host machine:

1) From the ESXi console, hit Alt-F1, then type "unsupported". You will get a bunch of dire warnings about this being an unsupported mode. You are now in a bare-bones Unix shell.

2) (optional) Enable ssh so you can do the rest remotely: use vi to edit the /etc/inetd.conf file and uncomment the line that starts with "ssh". Exit and restart inetd with "kill -HUP " where is the process ID of inetd. You can find the PID with "ps aux | grep inetd".

3) cd /vmfs/volumes/datastore

4) Use vmkfstools to clone the .vmdk file:

# vmkfstools -i imageA/imageA.vmdk imageB/imageB.vmdk
Destination disk format: VMFS thick
Cloning disk 'xubu1/xubu1.vmdk'...
Clone: 100% done.

5) From VI Client, create a new VM and select the custom option. When you get to the "select a hard disk" part, select the VMDK file you just cloned in the previous step.

6) You may have trouble with a cloned machine in Windows; you'll need to run sysprep to make it unique. In Linux you can just change the IP (if not using DHCP) and edit /etc/hostname and reboot to make a unique hostname.

steps to install Bro IDS on Ubuntu

This works on both Ubuntu and Xubuntu. Use sudo on everything or run as root.


apt-get install libncurses5-dev g++ bison flex
apt-get install libmagic-dev libgeoip-dev libpcap-dev libssl-dev
tar -xvf bro-1.4-release.tar.gz
cd bro-1.4
./configure --prefix=/usr/local/bro
make
make install

How to Install dig for Windows

dig is the standard tool for advanced DNS queries. A Windows version is available as part of the BIND port. To install it on Windows:

1) Go to ftp://ftp.isc.org/isc/bind9/9.5.0-P2/

2) Download BIND9.5.0-P2.zip

3) Open the archive with WinZip

4) Extract dig.exe and *.dll to c:\windows\system32

5) If you want the documentation page, extract dig.html to somewhere that you can find it.

Now you will be able to use dig from your command prompt in Windows. It is faster and more sophisticated than nslookup.

Get the quick help options with "dig -h".