port-map feature

Ever forget what port number maps to what service? A router running Adv IP Services, Adv Security, or Adv Enterprise Services will tell you all the common ones using the show ip port-map command, which is part of the IOS firewall feature set:

Router#sh ip port-map
Default mapping: snmp udp port 161 system defined
Default mapping: echo tcp port 7 system defined
Default mapping: echo udp port 7 system defined
Default mapping: telnet tcp port 23 system defined
Default mapping: wins tcp port 1512 system defined
Default mapping: n2h2server tcp port 9285 system defined
Default mapping: n2h2server udp port 9285 system defined
Default mapping: nntp tcp port 119 system defined
Default mapping: pptp tcp port 1723 system defined
Default mapping: rtsp tcp port 554,8554 system defined
Default mapping: bootpc udp port 68 system defined
Default mapping: gdoi udp port 848 system defined
Default mapping: tacacs udp port 49 system defined

[output truncated]

You can, of course, filter for stuff you find interesting:

Router#sh ip port-map | i 110
Default mapping: pop3 tcp port 110 system defined

in-line editing of Cisco ACLs

Many people don't realize that reasonably recent versions of IOS have in-line ACL editing.

The way this works is that ACLs have invisible line numbers that only show up when using a show access-list command, not when doing "show run":

R1#sh access-list TEST
Extended IP access list TEST
10 permit tcp host 1.1.1.1 host 2.2.2.2
20 permit gre host 3.3.3.3 host 4.4.4.4

In the example above, the line numbers are shown. If we just look at the config, they're not:

R1#sh run | s access-list
ip access-list extended TEST
permit tcp host 1.1.1.1 host 2.2.2.2
permit gre host 3.3.3.3 host 4.4.4.4


Now, let's say I want to make line number 10 more restrictive:


R1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)#ip access-list ext TEST
R1(config-ext-nacl)#15 permit tcp host 1.1.1.1 host 2.2.2.2 eq www
R1(config-ext-nacl)#do sh access-list TEST
Extended IP access list TEST
10 permit tcp host 1.1.1.1 host 2.2.2.2
15 permit tcp host 1.1.1.1 host 2.2.2.2 eq www
20 permit gre host 3.3.3.3 host 4.4.4.4
R1(config-ext-nacl)#no 10
R1(config-ext-nacl)#do sh access-list TEST
Extended IP access list TEST
15 permit tcp host 1.1.1.1 host 2.2.2.2 eq www
20 permit gre host 3.3.3.3 host 4.4.4.4


If the odd sequence numbers really bother you, you can fix them:

R1(config)#ip access-list resequence TEST ?
<1-2147483647> Starting Sequence Number

R1(config)#ip access-list resequence TEST 10 ?
<1-2147483647> Step to increment the sequence number

R1(config)#ip access-list resequence TEST 10 10 ?


R1(config)#ip access-list resequence TEST 10 10
R1(config)#do sh access-list TEST
Extended IP access list TEST
10 permit tcp host 1.1.1.1 host 2.2.2.2 eq www
20 permit gre host 3.3.3.3 host 4.4.4.4

I'm not sure what IOS version this appeared in, but it's been around since at least the early 12.3T images.

GRE tunnels and OSPF adjacencies

I made the mistake of starting to play with one of Ivan Pepelnjak's OSPF challenges on his great IOS blog, and got carried away playing with GRE tunnels instead.

On R1:

interface Loopback0
ip address 10.1.2.1 255.255.255.0

interface Serial1/0
ip unnumbered Loopback0
encapsulation ppp

interface Tunnel0
ip unnumbered Loopback0
ip ospf 1 area 1
tunnel source Serial1/0
tunnel destination 10.1.2.3



On R2:

interface Loopback0
ip address 2.2.2.2 255.255.255.0

interface Serial1/0
ip address 10.1.2.3 255.255.255.248
encapsulation ppp


interface Tunnel0
ip address 22.22.22.22 255.255.255.0
ip ospf 1 area 1
tunnel source Serial1/0
tunnel destination 10.1.2.1


This configuration forms no adjacency because the GRE tunnel interfaces are on different networks. A debug confirms this:


R2#deb ip os adj
OSPF adjacency events debugging is on
R2#
*Oct 29 09:43:03.123: OSPF: Rcv pkt from 10.1.2.1, Tunnel0, area 0.0.0.1 : src not on the same network

from a debug ip packet at the same time:

*Oct 29 10:08:11.015: IP: s=22.22.22.22 (Tunnel0), d=224.0.0.5, len 76, rcvd 0

[incidentally, I'm not sure why this behavior doesn't violate section 8.2 of RFC 2328, which states:

  ...the packet's IP source address is required to
be on the same network as the receiving interface.  This
can be verified by comparing the packet's IP source
address to the interface's IP address, after masking
both addresses with the interface mask.  This comparison
should not be performed on point-to-point networks. On
point-to-point networks, the interface addresses of each
end of the link are assigned independently, if they are
assigned at all.

GRE tunnels are considered point-to-point networks by OSPF.]

However, if I remove the explicitly configured IP address on R2's tunnel interface and replace it with an unnumbered configuration, using a loopback that's still not in the same network, it works:

R2(config)#int tu 0
R2(config-if)#ip unn lo 0
R2(config-if)#^Z
R2#
*Oct 29 09:44:44.907: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.1 on Tunnel0 from LOADING to FULL, Loading Done

R2#sh ip os n

Neighbor ID Pri State Dead Time Address Interface
10.0.0.1 0 FULL/ - 00:00:33 10.1.2.1 Tunnel0


R1#sh ip os n

Neighbor ID Pri State Dead Time Address Interface
10.1.2.3 0 FULL/ - 00:00:32 2.2.2.2 Tunnel0


At first I thought that maybe the router was changing the source address for OSPF packets to the address of the tunnel source, but that's not the case:

R1#deb ip packet
IP packet debugging is on
R1#
*Oct 29 10:05:44.955: IP: s=10.1.2.1 (local), d=224.0.0.5 (Tunnel0), len 80, sending broad/multicast
*Oct 29 10:05:44.955: IP: s=10.1.2.1 (Tunnel0), d=10.1.2.3 (Serial1/0), len 104, sending
*Oct 29 10:05:45.703: IP: s=2.2.2.2 (Tunnel0), d=224.0.0.5, len 80, rcvd 0

As you can see, the OSPF hellos are coming from the loopback IP of R2, which is on a completely different network.

IP phones show bogus "unknown" status in UCM

If you end up with a bunch of normally functioning, non-problematic IP phones showing up as "unknown" instead of "registered" in CallManager administration, you can restart the RIS Data Collector service on the publisher to solve the problem. This doesn't affect call processing, but it does seem to restart some media resources like the annunciator, etc.

certification musings

I started this blog intending it as a repository for odd network and VoIP related issues I run into over time, but I just have to vent a bit about a recent experience.

Last week I interviewed someone whose resume listed virtually every non-CCIE Cisco certification there is: CCNA, CCNP, CCSP, CCVP, CCDP, etc etc.

Now, I'm not naive enough to think that someone with all those certifications will necessarily be thoroughly proficient at all the subject areas (I have the CCSP among other things, but I'm pretty rusty on Cisco IDS), but a passing familiarity would be nice. This individual couldn't describe basic ARP operation, a basic TCP handshake, and couldn't name any layer 2 WAN protocols. He didn't do any better on voice or security questions.

Obviously it's possible to get all these certifications without knowing much, but if you're going to put them on your resume, at least try to be familiar with the basic concepts in the subject area.

Better yet, use the tests as a way to evaluate your REAL theoretical and practical knowledge of the subject area, not as something to spiff up your resume.

update on yesterday's application weirdness post

A friend read my post yesterday on odd problems with H.323 call setup on VT-Advantage-enabled phones after changing media resources configuration. He observed that IOS 12.4(20T), which we're running on the ISRs containing the DSPs, is the first IOS image to fully implement H.320 ISDN video conferencing capability, and that this might have something to do with the problem. The H.323 gateway that's actually handing the call to the PSTN isn't running 12.4(20T); it's running a much older image. Why the code on the DSP farm would affect a call on a different gateway, I don't understand, but it certainly seems like too much correlation to be coincidence.

Weird application interactions

Recently we made some changes to our media resource configuration in CallManager 6.x. Specifically, we stopped using DSPs on a 6608 blade in a Catalyst 6500 and started using DSPs in a couple of 2811 routers. Immediately after this change, a couple of users reported that they could no longer make outbound PSTN calls that transit a particular H.323 gateway. The weirdest part was that if they rebooted their 7940 phone, they could make a single call, but all subsequent calls would fail with a fast busy until they rebooted again. Calls through other gateways worked fine.

A ISDN Q.931 debug showed that the second outbound call was setting up as "unrestricted digital" instead of "speech".

I went 'round with TAC about this for a few emails and they were getting ready to blame the telco (?!) when I realized that the thing these two users have in common is VT Advantage (a Cisco desktop video-conferencing application). I asked them to turn off VT Advantage, and presto, all calls now worked.

Upgrading to the latest VT Advantage version solved the problem completely.

Why a change in media resources would affect outbound calling through a single H.323 gateway on users with an old version of VT Advantage is beyond me, but that's my story and I'm sticking to it.

Unity and Multiple Active Directory Domains

There seems to be a lack of information out there, even with some TAC engineers, about how to prepare Cisco Unity to import subscribers from Exchange servers in AD domains different from the one in which Unity is installed. I've even had several Cisco engineers tell me that this is unsupported, which is totally false. As far as I know, these are the requirements for Unity with respect to multiple domains:


1) All Exchange servers from which you want to import subscribers must be in the same AD forest. Unity unified messaging does not work across multiple forests. Of course, you can still create voicemail boxes that are totally independent of AD, but that's not the point of unified messaging. Exchange servers in different domains in the same forest is not a problem, as long as the PES (see below) is in the same domain as Unity.

2) The Unity server(s) must be installed in the same Active Directory domain as the partner Exchange server, or PES. You specify the PES during the Unity install process. It can be changed with the Message Store Integration Wizard. The PES must have Exchange routing group connectors to all other Exchange servers from which you want to import subscribers. More info on changing the PES is here.

3) You must run the Permissions Wizard on the Unity server while logged in as an account that can set permissions in all the domains containing Exchange servers from which you want to import subscribers. This might require running the PW multiple times under different accounts. This works fine. You may need to add a domain admin account from another domain to the local admin group on the Unity server. When you run PW from accounts in different domains, it will probably fail on some of the domains. This is because you probably won't have a single account that has all the required permissions in all the domains. You just need to continue running PW under different accounts until it has succeeded in all the domains that have Exchange servers from which you're importing subscribers.

Changes don't take effect in UCM 6

If you make a change to a Cisco Unified Communications Manager 6.x system and it doesn't seem to take effect, try restarting the Database Layer Monitor service and the TFTP service. This can be done during production hours.

In my case, the change in question was a change to the CallManager Group setting in a device pool.

Unity Permissions Wizard Stuff to Remember

To successfully run Unity Permission Wizard in a new domain, you need to run it as the Domain Admin in the remote domain. It will fail on the existing domains (the permissions are already set correctly on them, assuming things are already working), and succeed in the new domain.

Trivia Friday: ip subnet-zero

The ip subnet-zero command has been largely irrelevant since it became the default in IOS quite some time ago, but it still generates occasional discussion among consumers of IOS trivia.

For whatever reason, a lot of people seem to think that setting no ip subnet-zero causes you to be unable to use either the all-zeros OR all-ones subnets of a classful network. This is not the case. Turning off ip subnet-zero (not that you would want to do this outside of an IOS trivial pursuit game!) only prevents you from assigning the all-zeros subnet of a network:

R2(config)#no ip subnet-zero
R2(config)#int loopback 0
R2(config-if)#ip add 10.0.0.1 255.255.255.192
Bad mask /26 for address 10.0.0.1
R2(config-if)#ip add 10.255.255.254 255.255.255.192
R2(config-if)#ip add 192.168.0.1 255.255.255.192
Bad mask /26 for address 192.168.0.1
R2(config-if)#ip add 192.168.254.254 255.255.255.192
R2(config-if)#

Note that we are only prevented from assigning the all-zeros subnet of the network, and there's no restriction on assigning the all-ones subnet.

CallManager Trace Decoding Tools

Triple Combo looks to be a nice tool for decoding CCM/UCM trace files:


http://www.employees.org/~tiryaki/tc/

In the past I've also used TranslatorX, which is good but has some missing pieces:

http://www.employees.org/~pgiralt/TranslatorX/

changing IP address on Unity Express

Apparently you are supposed to shut down CUE before you change the IP address on the service module interface... I didn't, and the CUE module stopped accepting calls until it was reset with the "service module service-engine 0/1 reset" command.

troubleshooting basic POTS calls on a router

We get calls fairly regularly from users at sites with router-based VoIP systems (whether UCM Express or distributed UCM, it doesn't matter) saying that callers to their outside lines are getting ring-no-answer behavior.

The first thing I want to do is see if the calls are actually getting to the router. So I access the CLI and do the following:

Router#terminal monitor
Router#debug vpm signal

Then I place a test call to one of their outside lines. If the router just sits there and produces no output, then the problem is almost certainly either in the PSTN, or somebody has unplugged the cable from the FXO port.

If the router produces a bunch of barely comprehensible output, then the line is ringing and you've got further troubleshooting to do. Don't forget to turn off debugging when you're done!

DSCP Confusion

A friend of mine asked me a question the other day that pointed out a common area of confusion with the DiffServ model for IP QoS. He was reading some documentation that recommended setting the DSCP value for voice-signaling packets to 26 instead of 24. He was trying to convert those numbers to Assured Forwarding (AF) values, and came up with 31 and 30, respectively. His question was "Why does the documentation appear to be making the QoS treatment of these packets WORSE?"

The answer is, it's not. 30 is not a valid AF value. With the Assured Forwarding classes, there are three drop precedence values: 1, 2, and 3. Since zero is not a valid drop precedence, 30 is not a valid AF.

The DSCP value of 26 is 001000 in binary. The per-hop-behavior (PHB) value for this is actually not an Assured Forwarding PHB at all, it's a Class Selector value, CS3. The class selector values equate directly to the old IP precedence values, from the days before DiffServ. So DSCP 26 = CS3 = IP precedence 3 = 001000.

Clear as mud? Try the Wikipedia article.

IPv6 at IETF

A few weeks ago at NANOG they turned off IPv4 on the wireless network for a while. Here's a story about the same thing at the much larger IETF meeting.

MBXSuite tool for troubleshooting Unity mailbox issues

I recently learned about a useful tool for troubleshooting Unity/Exchange mailbox issues. It's located in the \CommServer\TechTools folder, and it's called MBXSuite.exe.

To use it:

1) Select the user whose mailbox you want to test with.
2) Select the Unity service account you want to run as--usually UnityMsgStoreSvc.
3) Check "Verbose Diags"
4) Click "Logon Mailbox" and check the Diagnostics pane for errors.