RIP database and administrative distance

I was helping a friend study for CCNA the other day and saw a RIP behavior I'd never noticed before. I knew that RIP keeps a local route database that is displayed with the show ip rip database command. If another route to the same prefix with a better administrative distance is preferred in the global routing table, however, the RIP database doesn't show the route. This is different than more sophisticated routing protocols in which a prefix is kept in the protocol-specific topology table even if a route from another protocol with a better AD is in the global routing table.

Cisco IPS Manager Express

I've been doing Cisco IDS/IPS stuff recently for the first time in a long while. If you haven't tried Cisco's new free IPS Manager Express application, check it out. It makes IDS/IPS event monitoring and management reasonably useful and almost pain-free. The interface is much more intuitive that other Cisco IDS/IPS GUI products. The only problem is that the current version supports only 5 sensors; supposedly this will increase in a future release.

Added 12/15/09:
The latest version of IME supports 10 sensors.

open ports on IOS router

Haven't posted here in ages. Interesting trivia: the old "show ip sockets" command doesn't work in new 12.4T images. It's been replaced by "show control-plane host open-ports":

#sh control-plane host open-ports
Active internet connections (servers and established)
Prot Local Address Foreign Address Service State
tcp *:22 *:0 SSH-Server LISTEN
tcp *:23 *:0 Telnet LISTEN
tcp *:15904 x.x.x.x:179 IOS host service ESTABLIS
tcp *:179 x.x.x.x:38441 BGP ESTABLIS
tcp *:179 *:0 BGP LISTEN
tcp *:179 *:0 BGP LISTEN
tcp *:179 *:0 BGP LISTEN
udp *:49 x.x.x.x:0 TACACS service LISTEN
udp *:161 *:0 IP SNMP LISTEN
udp *:162 *:0 IP SNMP LISTEN
udp *:57421 *:0 IP SNMP LISTEN
udp *:1985 *:0 cisco HSRP LISTEN

how to clone a VM in the free VMWare ESXi

In the free verison of ESXi, it's not obvious how to clone a VM, since you don't have VirtualCenter available.

I've read about some people using the free version of Converter, but this didn't work for me (Converter keeps hanging partway through the operation). Here's what I did; note that I'm running local storage only on an older host machine:

1) From the ESXi console, hit Alt-F1, then type "unsupported". You will get a bunch of dire warnings about this being an unsupported mode. You are now in a bare-bones Unix shell.

2) (optional) Enable ssh so you can do the rest remotely: use vi to edit the /etc/inetd.conf file and uncomment the line that starts with "ssh". Exit and restart inetd with "kill -HUP " where is the process ID of inetd. You can find the PID with "ps aux | grep inetd".

3) cd /vmfs/volumes/datastore

4) Use vmkfstools to clone the .vmdk file:

# vmkfstools -i imageA/imageA.vmdk imageB/imageB.vmdk
Destination disk format: VMFS thick
Cloning disk 'xubu1/xubu1.vmdk'...
Clone: 100% done.

5) From VI Client, create a new VM and select the custom option. When you get to the "select a hard disk" part, select the VMDK file you just cloned in the previous step.

6) You may have trouble with a cloned machine in Windows; you'll need to run sysprep to make it unique. In Linux you can just change the IP (if not using DHCP) and edit /etc/hostname and reboot to make a unique hostname.

steps to install Bro IDS on Ubuntu

This works on both Ubuntu and Xubuntu. Use sudo on everything or run as root.


apt-get install libncurses5-dev g++ bison flex
apt-get install libmagic-dev libgeoip-dev libpcap-dev libssl-dev
tar -xvf bro-1.4-release.tar.gz
cd bro-1.4
./configure --prefix=/usr/local/bro
make
make install

How to Install dig for Windows

dig is the standard tool for advanced DNS queries. A Windows version is available as part of the BIND port. To install it on Windows:

1) Go to ftp://ftp.isc.org/isc/bind9/9.5.0-P2/

2) Download BIND9.5.0-P2.zip

3) Open the archive with WinZip

4) Extract dig.exe and *.dll to c:\windows\system32

5) If you want the documentation page, extract dig.html to somewhere that you can find it.

Now you will be able to use dig from your command prompt in Windows. It is faster and more sophisticated than nslookup.

Get the quick help options with "dig -h".

port-map feature

Ever forget what port number maps to what service? A router running Adv IP Services, Adv Security, or Adv Enterprise Services will tell you all the common ones using the show ip port-map command, which is part of the IOS firewall feature set:

Router#sh ip port-map
Default mapping: snmp udp port 161 system defined
Default mapping: echo tcp port 7 system defined
Default mapping: echo udp port 7 system defined
Default mapping: telnet tcp port 23 system defined
Default mapping: wins tcp port 1512 system defined
Default mapping: n2h2server tcp port 9285 system defined
Default mapping: n2h2server udp port 9285 system defined
Default mapping: nntp tcp port 119 system defined
Default mapping: pptp tcp port 1723 system defined
Default mapping: rtsp tcp port 554,8554 system defined
Default mapping: bootpc udp port 68 system defined
Default mapping: gdoi udp port 848 system defined
Default mapping: tacacs udp port 49 system defined

[output truncated]

You can, of course, filter for stuff you find interesting:

Router#sh ip port-map | i 110
Default mapping: pop3 tcp port 110 system defined

in-line editing of Cisco ACLs

Many people don't realize that reasonably recent versions of IOS have in-line ACL editing.

The way this works is that ACLs have invisible line numbers that only show up when using a show access-list command, not when doing "show run":

R1#sh access-list TEST
Extended IP access list TEST
10 permit tcp host 1.1.1.1 host 2.2.2.2
20 permit gre host 3.3.3.3 host 4.4.4.4

In the example above, the line numbers are shown. If we just look at the config, they're not:

R1#sh run | s access-list
ip access-list extended TEST
permit tcp host 1.1.1.1 host 2.2.2.2
permit gre host 3.3.3.3 host 4.4.4.4


Now, let's say I want to make line number 10 more restrictive:


R1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)#ip access-list ext TEST
R1(config-ext-nacl)#15 permit tcp host 1.1.1.1 host 2.2.2.2 eq www
R1(config-ext-nacl)#do sh access-list TEST
Extended IP access list TEST
10 permit tcp host 1.1.1.1 host 2.2.2.2
15 permit tcp host 1.1.1.1 host 2.2.2.2 eq www
20 permit gre host 3.3.3.3 host 4.4.4.4
R1(config-ext-nacl)#no 10
R1(config-ext-nacl)#do sh access-list TEST
Extended IP access list TEST
15 permit tcp host 1.1.1.1 host 2.2.2.2 eq www
20 permit gre host 3.3.3.3 host 4.4.4.4


If the odd sequence numbers really bother you, you can fix them:

R1(config)#ip access-list resequence TEST ?
<1-2147483647> Starting Sequence Number

R1(config)#ip access-list resequence TEST 10 ?
<1-2147483647> Step to increment the sequence number

R1(config)#ip access-list resequence TEST 10 10 ?


R1(config)#ip access-list resequence TEST 10 10
R1(config)#do sh access-list TEST
Extended IP access list TEST
10 permit tcp host 1.1.1.1 host 2.2.2.2 eq www
20 permit gre host 3.3.3.3 host 4.4.4.4

I'm not sure what IOS version this appeared in, but it's been around since at least the early 12.3T images.

GRE tunnels and OSPF adjacencies

I made the mistake of starting to play with one of Ivan Pepelnjak's OSPF challenges on his great IOS blog, and got carried away playing with GRE tunnels instead.

On R1:

interface Loopback0
ip address 10.1.2.1 255.255.255.0

interface Serial1/0
ip unnumbered Loopback0
encapsulation ppp

interface Tunnel0
ip unnumbered Loopback0
ip ospf 1 area 1
tunnel source Serial1/0
tunnel destination 10.1.2.3



On R2:

interface Loopback0
ip address 2.2.2.2 255.255.255.0

interface Serial1/0
ip address 10.1.2.3 255.255.255.248
encapsulation ppp


interface Tunnel0
ip address 22.22.22.22 255.255.255.0
ip ospf 1 area 1
tunnel source Serial1/0
tunnel destination 10.1.2.1


This configuration forms no adjacency because the GRE tunnel interfaces are on different networks. A debug confirms this:


R2#deb ip os adj
OSPF adjacency events debugging is on
R2#
*Oct 29 09:43:03.123: OSPF: Rcv pkt from 10.1.2.1, Tunnel0, area 0.0.0.1 : src not on the same network

from a debug ip packet at the same time:

*Oct 29 10:08:11.015: IP: s=22.22.22.22 (Tunnel0), d=224.0.0.5, len 76, rcvd 0

[incidentally, I'm not sure why this behavior doesn't violate section 8.2 of RFC 2328, which states:

  ...the packet's IP source address is required to
be on the same network as the receiving interface.  This
can be verified by comparing the packet's IP source
address to the interface's IP address, after masking
both addresses with the interface mask.  This comparison
should not be performed on point-to-point networks. On
point-to-point networks, the interface addresses of each
end of the link are assigned independently, if they are
assigned at all.

GRE tunnels are considered point-to-point networks by OSPF.]

However, if I remove the explicitly configured IP address on R2's tunnel interface and replace it with an unnumbered configuration, using a loopback that's still not in the same network, it works:

R2(config)#int tu 0
R2(config-if)#ip unn lo 0
R2(config-if)#^Z
R2#
*Oct 29 09:44:44.907: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.1 on Tunnel0 from LOADING to FULL, Loading Done

R2#sh ip os n

Neighbor ID Pri State Dead Time Address Interface
10.0.0.1 0 FULL/ - 00:00:33 10.1.2.1 Tunnel0


R1#sh ip os n

Neighbor ID Pri State Dead Time Address Interface
10.1.2.3 0 FULL/ - 00:00:32 2.2.2.2 Tunnel0


At first I thought that maybe the router was changing the source address for OSPF packets to the address of the tunnel source, but that's not the case:

R1#deb ip packet
IP packet debugging is on
R1#
*Oct 29 10:05:44.955: IP: s=10.1.2.1 (local), d=224.0.0.5 (Tunnel0), len 80, sending broad/multicast
*Oct 29 10:05:44.955: IP: s=10.1.2.1 (Tunnel0), d=10.1.2.3 (Serial1/0), len 104, sending
*Oct 29 10:05:45.703: IP: s=2.2.2.2 (Tunnel0), d=224.0.0.5, len 80, rcvd 0

As you can see, the OSPF hellos are coming from the loopback IP of R2, which is on a completely different network.

IP phones show bogus "unknown" status in UCM

If you end up with a bunch of normally functioning, non-problematic IP phones showing up as "unknown" instead of "registered" in CallManager administration, you can restart the RIS Data Collector service on the publisher to solve the problem. This doesn't affect call processing, but it does seem to restart some media resources like the annunciator, etc.

certification musings

I started this blog intending it as a repository for odd network and VoIP related issues I run into over time, but I just have to vent a bit about a recent experience.

Last week I interviewed someone whose resume listed virtually every non-CCIE Cisco certification there is: CCNA, CCNP, CCSP, CCVP, CCDP, etc etc.

Now, I'm not naive enough to think that someone with all those certifications will necessarily be thoroughly proficient at all the subject areas (I have the CCSP among other things, but I'm pretty rusty on Cisco IDS), but a passing familiarity would be nice. This individual couldn't describe basic ARP operation, a basic TCP handshake, and couldn't name any layer 2 WAN protocols. He didn't do any better on voice or security questions.

Obviously it's possible to get all these certifications without knowing much, but if you're going to put them on your resume, at least try to be familiar with the basic concepts in the subject area.

Better yet, use the tests as a way to evaluate your REAL theoretical and practical knowledge of the subject area, not as something to spiff up your resume.

update on yesterday's application weirdness post

A friend read my post yesterday on odd problems with H.323 call setup on VT-Advantage-enabled phones after changing media resources configuration. He observed that IOS 12.4(20T), which we're running on the ISRs containing the DSPs, is the first IOS image to fully implement H.320 ISDN video conferencing capability, and that this might have something to do with the problem. The H.323 gateway that's actually handing the call to the PSTN isn't running 12.4(20T); it's running a much older image. Why the code on the DSP farm would affect a call on a different gateway, I don't understand, but it certainly seems like too much correlation to be coincidence.

Weird application interactions

Recently we made some changes to our media resource configuration in CallManager 6.x. Specifically, we stopped using DSPs on a 6608 blade in a Catalyst 6500 and started using DSPs in a couple of 2811 routers. Immediately after this change, a couple of users reported that they could no longer make outbound PSTN calls that transit a particular H.323 gateway. The weirdest part was that if they rebooted their 7940 phone, they could make a single call, but all subsequent calls would fail with a fast busy until they rebooted again. Calls through other gateways worked fine.

A ISDN Q.931 debug showed that the second outbound call was setting up as "unrestricted digital" instead of "speech".

I went 'round with TAC about this for a few emails and they were getting ready to blame the telco (?!) when I realized that the thing these two users have in common is VT Advantage (a Cisco desktop video-conferencing application). I asked them to turn off VT Advantage, and presto, all calls now worked.

Upgrading to the latest VT Advantage version solved the problem completely.

Why a change in media resources would affect outbound calling through a single H.323 gateway on users with an old version of VT Advantage is beyond me, but that's my story and I'm sticking to it.

Unity and Multiple Active Directory Domains

There seems to be a lack of information out there, even with some TAC engineers, about how to prepare Cisco Unity to import subscribers from Exchange servers in AD domains different from the one in which Unity is installed. I've even had several Cisco engineers tell me that this is unsupported, which is totally false. As far as I know, these are the requirements for Unity with respect to multiple domains:


1) All Exchange servers from which you want to import subscribers must be in the same AD forest. Unity unified messaging does not work across multiple forests. Of course, you can still create voicemail boxes that are totally independent of AD, but that's not the point of unified messaging. Exchange servers in different domains in the same forest is not a problem, as long as the PES (see below) is in the same domain as Unity.

2) The Unity server(s) must be installed in the same Active Directory domain as the partner Exchange server, or PES. You specify the PES during the Unity install process. It can be changed with the Message Store Integration Wizard. The PES must have Exchange routing group connectors to all other Exchange servers from which you want to import subscribers. More info on changing the PES is here.

3) You must run the Permissions Wizard on the Unity server while logged in as an account that can set permissions in all the domains containing Exchange servers from which you want to import subscribers. This might require running the PW multiple times under different accounts. This works fine. You may need to add a domain admin account from another domain to the local admin group on the Unity server. When you run PW from accounts in different domains, it will probably fail on some of the domains. This is because you probably won't have a single account that has all the required permissions in all the domains. You just need to continue running PW under different accounts until it has succeeded in all the domains that have Exchange servers from which you're importing subscribers.