Friday, April 24, 2015

ADN - Awk Defined Networking

Because I have yet to transition to a completely software-defined network in which everything configures itself (wink wink), I still have to do tasks like bulk VLAN changes.

Thanks to a recent innovation called ADN, or "AWK Defined Networking", I can do this in a shorter time window that the average bathroom break. For example, I just had a request to change all ports on a large access switch stack that  are currently in VLAN 76 to VLAN 64:

# ssh switch_name.foo.com 'show int status | i _76_' | grep Gi | awk '{print "int ",$1,"\n","description PC/Phone","\n","switchport access vlan 64"}'
Password: ***


int  Gi1/0/25
 description PC/Phone
 switchport access vlan 64
int  Gi1/0/26
 description PC/Phone
 switchport access vlan 64

[many more deleted]

Then I copied and pasted the results into config mode. Back to lounging on the beach.

Not even any Python skills required!

Thursday, March 26, 2015

Quick Example: Elasticsearch Bulk Index API with Python

A quick example that shows how to use Elasticsearch bulk indexing from the Python client. This is dramatically faster than indexing documents one at a time in a loop with the index() method.

Saturday, February 28, 2015

Filtering .raw fields with Python Elasticsearch DSL High-Level Client

It took me a while to figure out how to search the not_analyzed ".raw" fields created by Logstash in Elasticsearch indices, using the high-level Python Elasticsearch client. Because keyword arguments can't have attributes, Python throws an error if you try it the intuitive way (this assumes you've already set up a client as es and an index as i, as shown in the docs):

Instead, you create a dictionary with your parameters and unpack it using the ** operator:

This produces the Elasticsearch query we want:

Thursday, February 12, 2015

Experimenting with the CSR1000v REST API

This all started because we occasionally want to block traffic from an IP address or two for a short time. Our firewall is a pain to configure for this sort of thing: adding a drop for a single IP address literally takes 10 minutes. You have to open a fat client, create an object, add the object to a group, save the config, verify the config, push the config, etc.

I thought that SRTBH (Source-based Real-Time Black Hole) implemented by BGP would be the ticket: fast, easy, and theoretically easy to automate with the REST API in the Cisco Cloud Services Router 1000v. SRTBH is a simple and elegant way of dropping selected traffic on BGP speaking routers. In a nutshell:
  1. You configure a "trigger router" that speaks iBGP with the rest of your BGP-speaking routers (usually your Internet edge or transit routers), but doesn't participate in traffic forwarding.
  2. On each edge/transit router you configure a static route to null0 for an unused /32, usually 192.0.2.1: ip route 192.0.2.1 255.255.255.255 null0
  3. On each edge/transit router you configure loose-mode unicast RPF filtering on your outside interfaces: ip verify source reachable-via any. This has a special property: packets from any source for which the uRPF next-hop resolves to Null0 will be dropped.
  4. When you want to drop traffic from a particular host (e.g. 1.1.1.2), you configure a static host route on the trigger router and redistribute it into BGP by matching its tag: ip route 1.1.1.2 255.255.255.255 null0 tag 666.
  5. At redistribution, you set its next-hop to 192.0.2.1:

    route-map SRTBH permit 10
     description BGP Black Hole
     match tag 666
     set origin igp
     set community no-export
     set ip next-hop 192.0.2.1
The effect of this is that the edge routers see the iBGP route to the "bad" address as a /32 with a next-hop of 192.0.2.1. Because the next-hop for 192.0.2.1 recursively resolves to null0, the loose uRPF feature drops traffic from the bad host at the edge.

In my case, I wanted to use the CSR1000v as the trigger router. (At this point, somebody's thinking "Why not use a Linux BGP daemon?") That's a valid approach; I just want to use the CSR1000v.

I followed the REST setup instructions, then wrote a simple CLI tool to make testing easier. It's easy enough to test the API with cURL:

$ curl -X POST https://10.1.1.1:55443/api/v1/auth/token-services -H "Accept:application/json" -u "testuser" -d "" --insecure -3
Enter host password for user 'testuser':
{"kind": "object#auth-token", "expiry-time": "Thu Feb 12 17:24:30 2015", "token-id": "xNOvrJh[...]rW3x0=", "link": "https://10.1.1.1:55443/api/v1/auth/token-services/2285580579"}


but because the API requires that you obtain a temporary authorization token (demonstrated above) before doing anything interesting, writing the Python tool made sense.

Then the problems started: the REST API on the CSR1000v doesn't support tags on static routes. You can't add them, or even retrieve them. Here's what happens normally if you retrieve static routes:

>python rest.py --device 10.1.1.1 --user testuser --resource /routing-svc/static-routes
Password:
{u'items': [{u'admin-distance': 1,
             u'destination-network': u'10.0.0.0/8',
             u'kind': u'object#static-route',
             u'next-hop-router': u'10.30.254.152',
             u'outgoing-interface': u''},
 u'kind': u'collection#static-route'}


If one of the routes has a tag, it throws an error:

{u'detail': u' ',
 u'error-code': -1,
 u'error-message': u"invalid literal for int() with base 10: 'tag'"}


You can't set tags either. Furthermore, the same thing holds true for static routes with the "name" argument. (Incidentally, the error message seems to indicate that the IOS XE API is written in Python.)

The problem with this is that without tags, there's no convenient way to filter the RTBH black hole trigger routes at the time of redistribution. You can't set route-maps on BGP network statements via the API either, so no joy there.

The best option to me seems to be to redistribute all static routes, but filter on prefix-length rather than tag:

route-map SRTBH permit 10
 description BGP Black Hole
 match ip address prefix-list HOST_ROUTES_ONLY
 set origin igp
 set community no-export
 set ip next-hop 192.0.2.1


But then you have to make sure that you never have any /32s that aren't trigger routes... a strategy which is prone to configuration mistakes.

Because SRTBH seems like a natural use-case for API automation, I hope Cisco fixes this soon.

[added]
Other interesting things I've found: the static route configuration API doesn't allow you to add equal-cost static routes. For example, if I already have a 1.1.1.5/32 route and I add another one, I get a 404:

python rest.py --device 10.1.1.1 --user testuser --password foo --resource /routing-svc/static-routes --method post --json '{"destination-network":"1.1.1.5/32","outgoing-interface":"null0"}' --verbose
[snip]
DEBUG:root:got status code: 404
DEBUG:root:reponse content: Static route already exists
got status code: 404


The same thing goes for adding other items: I tried adding a name-server twice, and got a similar error. I would think that the API would be idempotent where possible (such as when adding a duplicate nameserver), and require clarifying attributes otherwise. With equal-cost static routes for example, it could throw a 40x unless an extra attribute like {"allow-equal-cost":true} is present in the POST request.

Thursday, January 15, 2015

Pleasing terminal colors on Security Onion

To get the lovely Solarized theme working in Security Onion:
  1. sudo apt-get install gnome-terminal
    1. I'm sure there's a way to get in working in the default xfce4 terminal, but I couldn't figure it out.
  2. Follow instructions here: http://stackoverflow.com/questions/23118916/configuring-solarized-colorscheme-in-gnome-terminal-tmux-and-vim

Thursday, January 8, 2015

Problems with kvm-ok in VIRL with VMWare Player

I'm installing Cisco VIRL, and despite following the instructions regarding nested virtualization settings, the kvm-ok command was still complaining. I needed to edit the .vmx file for the VIRL VM and add/edit the following:

monitor.virtual_mmu = "hardware"
monitor.virtual_exec = "hardware"
vhv.enable = "TRUE"
monitor_control.restrict_backdoor = "true"
 
 

Friday, January 2, 2015

My Network Toolkit

A while back, Chris Marget of Fragmentation Needed posted a run-down of his comprehensive and extremely clever network toolkit. Because I'm something of a weight weenie, mine is a lot more slimmed down. I thought I'd post it here:




The contents:

  1. Two random USB drives (in case I need to leave one with somebody).
  2. Single-mode and multi-mode LC fiber loopback plugs.
  3. Rack PDU plug adapter.
  4. Awesome PicQuic compact screwdriver (thanks to Chris's post).
  5. T1 loopback plug (red) (because we still have T1s out here in the boonies).
  6. Cat-6 pass-through plug (white).
  7. Crossover adapter (orange).
  8. Sharpie.
  9. Console setup:
    1. USB-to-DB9 adapter.
    2. DB9-to-RJ45 adapter.
    3. Flat Cat-6 cable.
    4. Rollover adapter.
    5. Velcro tie
    6. Flat Cat-6 cable with velcro tie.
The console setup could probably be improved by adding a DB9 null-modem adapter. The coolest thing (IMO) that I'm missing from Chris's setup is the Bluetooth console adapter -- maybe one day.

The Fenix AA light and Leatherman Skeletool CX almost always live in a pocket rather than the kit and go with me everywhere. The kit all fits into a small zippered case that used to hold a Dell laptop power supply.

My main goal here was to have all the hard-to-find professional stuff in one small package. I have a separate "personal" kit that contains stuff like headphones, USB cables, and chargers for personal electronics.

Friday, December 5, 2014

Imposing Artificial Limitations to Develop Skills

I'm a big fan of imposing artificial limitations on yourself in order to aid skill development. Here are some quick ideas:

  • When troubleshooting network devices from the CLI, try not to look at the configuration. Use only "show" or "debug" commands instead. I found this enormously beneficial when practicing for CCIE.
  • When troubleshooting larger operational issues or learning a new environment, try not to log into individual devices at all. Force yourself to use only your network management system, NetFlow, packet captures, or host-based tools like ping, traceroute, or nmap.
  • When learning automation or orchestration skills, force yourself to write scripts, run API calls, or use your favorite orchestration tool to do simple things, even if it doesn't seem like they merit the extra effort.

Tuesday, July 1, 2014

Simple Python Syslog Counter

Recently I did a Packet Pushers episode about log management. In it, I mentioned some of the custom Python scripts that I run to do basic syslog analysis, and someone asked about them in the comments.

The script I'm presenting here isn't one of the actual ones that I run in production, but it's close. The real one sends emails, does DNS lookups, keeps a "rare messages" database using sqlite3, and a few other things, but I wanted to keep this simple.

One of the problems I see with getting started with log analysis is that people tend to approach it like a typical vendor RFP project: list some requirements, survey the market, evaluate and buy a product to fit your requirements. Sounds good, right? The problem with log analysis is that often you don't know what your requirements really are until you start looking at data.

A simple message counting script like this lets you look at your data, and provides a simple platform on which you can start to iterate to find your specific needs. It also lets us look at some cool Python features.

I don't recommend pushing this too far: once you have a decent idea of what your data looks like and what you want to do with it, set up Logstash, Graylog2, or a similar commercial product like Splunk (if you can afford it).

That said, here's the Python:


I tried to make this as self-documenting as possible. You run it from the CLI with a syslog file as the argument, and you get this:

$ python simple_syslog_count.py sample.txt
 214   SEC-6-IPACCESSLOGP
 15    SEC-6-IPACCESSLOGRL
 10    LINEPROTO-5-UPDOWN
 10    LINK-3-UPDOWN
 7     USER-3-SYSTEM_MSG
 4     STACKMGR-4-STACK_LINK_CHANGE
 4     DUAL-5-NBRCHANGE
 3     IPPHONE-6-UNREGISTER_NORMAL
 3     CRYPTO-4-PKT_REPLAY_ERR
 3     SEC-6-IPACCESSLOGRP
 3     SEC-6-IPACCESSLOGSP
 2     SSH-5-SSH2_USERAUTH
 2     SSH-5-SSH2_SESSION
 2     SSH-5-SSH2_CLOSE

10.1.16.12

     6     SEC-6-IPACCESSLOGP

10.1.24.3

     2     LINEPROTO-5-UPDOWN
     2     LINK-3-UPDOWN


[Stuff deleted for brevity]

For Pythonistas, the script makes use of a few cool language features:

Named, Compiled rRgexes

  • We can name a regex match with the (?PPATTERN) syntax, which makes it easy to understand it when it's referenced later with the .group('') method on the match object.
  • This is demonstrated in lines 36-39 and 58-59 of the gist shown above. 
  • It would be more efficient to capture these fields by splitting the line with the .split() string method, but I wanted the script to work for unknown field positions -- hence the regex. 

Multiplication of Strings

  • We control indentation by multiplying the ' ' string (that a single space enclosed in quotes) by an integer value in the print_counter function (line 50).
    • The reason this works is that the Python str class defines a special __mul__ method that controls how the * operator works for objects of that class:
      >>> 'foo'.__mul__(3)
      'foofoofoo'
      >>> 'foo' * 3
      'foofoofoo'

collections.Counter Objects

  • Counter objects are a subclass of dictionaries that know how to count things. Jeremy Schulman talked about these in a comment on the previous post. Here, we use Counters to build both the overall message counts and the per-device message counts:
>>> my_msg = 'timestamp ip_address stuff %MY-4-MESSAGE:other stuff'
>>> CISCO_MSG = re.compile('%(?P.*?):')
>>> from collections import Counter
>>> test_counter = Counter()
>>> this_msg = re.search(CISCO_MSG,my_msg).group('msg')
>>> this_msg
'MY-4-MESSAGE'
>>> test_counter[this_msg] += 1
>>> test_counter
Counter({'MY-4-MESSAGE': 1})

collections.defaultdict Dictionaries

  • It could get annoying when you're assigning dictionary values inside a loop, because you get errors when the key doesn't exist yet. This is a contrived example, but it illustrates the point:

    >>> reporters = {}
    >>> for reporter in ['1.1.1.1','2.2.2.2']:
    ...     reporters[reporter].append['foo']
    ...
    Traceback (most recent call last):
      File "", line 2, in
    KeyError: '1.1.1.1'

     
  • To fix this, you can catch the exception:

    >>> reporters = {}
    >>> for reporter in ['1.1.1.1','2.2.2.2']:
    ...     try:
    ...         reporters[reporter].append['foo']
    ...         reporters[reporter].append['bar']
    ...     except KeyError:
    ...         reporters[reporter] = ['foo']
    ...         reporters[reporter].append('bar')
  • As usual, though, Python has a more elegant way in the collections module: defaultdict
>>> from collections import defaultdict
>>> reporters = defaultdict(list)
>>> for reporter in ['1.1.1.1','2.2.2.2']:
...     reporters[reporter].append('foo')
...     reporters[reporter].append('bar')
>>> reporters
defaultdict(, {'1.1.1.1': ['foo', 'bar'], '2.2.2.2': ['foo', 'bar']})
In the syslog counter script, we use a collections.Counter object as the type for our defaultdict. This allows us to build a per-syslog-reporter dictionary that shows how many times each message appears for each reporter, while only looping through the input once (line 66):

 per_reporter_counts[reporter][msg] += 1

Here, the dictionary per_reporter_counts has the IPv4 addresses of the syslog reporters as keys, with a Counter object as the value holding the counts for each message type:

>>> from collections import Counter,defaultdict
>>> per_reporter_counts = defaultdict(Counter)
>>> per_reporter_counts['1.1.1.1']['SOME-5-MESSAGE'] += 1
>>> per_reporter_counts
defaultdict(, {'1.1.1.1': Counter({'SOME-5-MESSAGE': 1})})
>>> per_reporter_counts['1.1.1.1']['SOME-5-MESSAGE'] += 5
>>> per_reporter_counts
defaultdict(, {'1.1.1.1': Counter({'SOME-5-MESSAGE': 6})})


If you got this far, you can go implement it for IPv6 addresses. :-)

Friday, June 20, 2014

Python Sets: Handy for Network Data

My Python-related posts seem to get the most reads, so here's another one!

A problem that comes up fairly often in networking is finding the number of occurrences of unique items in a large collection of data: let's say you want to find all of the unique IP addresses that accessed a website, traversed a firewall, got denied by an ACL, or whatever. Maybe you've extracted the following list from a log file:

1.1.1.1
2.2.2.2
3.3.3.3
1.1.1.1
5.5.5.5
5.5.5.5
1.1.1.1
2.2.2.2
...

and you need to reduce this to:

1.1.1.1
2.2.2.2
3.3.3.3
5.5.5.5

In other words, we're removing the duplicates. In low-level programming languages, removing duplicates is a bit of a pain: generally you need to implement an efficient way to sort an array of items, then traverse the sorted array to check for adjacent duplicates and remove them. In a language that has dictionaries (also known as hash tables or associative arrays), you can do it by adding each item as a key in your dictionary with an empty value, then extract the keys. In Python:

>>> items = ['1.1.1.1','2.2.2.2','3.3.3.3','1.1.1.1','5.5.5.5','5.5.5.5','1.1.1.1','2.2.2.2']
>>> d = {}
>>> for item in items:
...     d[item] = None
...
>>> d
{'5.5.5.5': None, '3.3.3.3': None, '1.1.1.1': None, '2.2.2.2': None}
>>> unique = d.keys()
>>> unique
['5.5.5.5', '3.3.3.3', '1.1.1.1', '2.2.2.2']


or, more concisely using a dictionary comprehension:

>>> {item:None for item in items}.keys()
['5.5.5.5', '3.3.3.3', '1.1.1.1', '2.2.2.2']


Python has an even better way, however: the "set" type, which emulates the mathematical idea of a set as a collection of distinct items. If you create an empty set and add items to it, duplicates will automatically be thrown away:

>>> s = set()
>>> s.add('1.1.1.1')
>>> s
set(['1.1.1.1'])
>>> s.add('2.2.2.2')
>>> s.add('1.1.1.1')
>>> s
set(['1.1.1.1', '2.2.2.2'])
>>> for item in items:
...     s.add(item)
...
>>> s
set(['5.5.5.5', '3.3.3.3', '1.1.1.1', '2.2.2.2'])


Predictably, you can use set comprehensions just like list comprehensions to do the same thing as a one liner:

>>> {item for item in items}
set(['5.5.5.5', '3.3.3.3', '1.1.1.1', '2.2.2.2'])


Or, if you have a list built already you can just convert it to a set:

>>> set(items)
set(['5.5.5.5', '3.3.3.3', '1.1.1.1', '2.2.2.2'])


Python also provides methods for the most common types of set operations: union, intersection, difference and symmetric difference. Because these methods accept lists or other iterables, you can quickly find similarities between collections of items:

>>> items
['1.1.1.1', '2.2.2.2', '3.3.3.3', '1.1.1.1', '5.5.5.5', '5.5.5.5', '1.1.1.1', '2.2.2.2']
>>> more_items = ['1.1.1.1','8.8.8.8','1.1.1.1','7.7.7.7','2.2.2.2']
>>> set(items).intersection(more_items)
set(['1.1.1.1', '2.2.2.2'])

>>> set(items).difference(more_items)
set(['5.5.5.5', '3.3.3.3'])


Have fun!

Wednesday, April 2, 2014

Fun with Router IP Traffic Export and NSM

The Basics
I finally got around to setting up Security Onion (the best network security monitoring package available) to monitor my home network, only to discover that my Cisco 891 router doesn't support support the right form of SPAN. Here's how I worked around it. The topology looks like this:


The 891 router has an integrated 8-port switch module, so the simple case would have been a traditional SPAN setup; something like this:

! vlan 10 is the user VLAN
monitor session 1 source interface vlan 10
monitor session 1 destination interface FastEthernet0

with the server's monitoring NIC connected to FastEthernet0.

The problem is that the 891 doesn't support using a VLAN as a source interface, and because of the way the embedded WAP works, a physical source interface won't work either. Hence, I turned to an obscure feature that's helped me occasionally in the past: Router IP Traffic Export. This is a feature for IOS software platforms that enables you to enable SPAN-like functions for almost any source interface.

The configuration looks like this:

ip traffic-export profile RITE_MIRROR
  interface FastEthernet0
  bidirectional
  mac-address 6805.ca21.2ddd


interface Vlan10
 ip traffic-export apply RITE_MIRROR

This takes all traffic routed across the Vlan10 SVI and sends it out the FastEthernet0 interface, rewriting the destination MAC address to the specified value. I used the MAC address of my monitoring NIC, but it shouldn't matter in this case because the monitoring NIC is directly attached. If I wanted to copy the traffic across a switched interface, it would matter.

My ESXi host (a low-cost machine from Zareason with 32GB of RAM) has two physical NICs; one for all of the regular VM traffic (using 802.1q to separate VLANs if needed) and one for monitoring. The monitoring pNIC is attached to a promiscuous mode vSwitch in ESXi, which in turn is connected to the monitoring vNIC on the Security Onion VM. The effect of this is identical to SPAN-ing all the traffic from VLAN 10 to my Security Onion monitoring system; I get Snort, Bro, Argus, and full packet capture with just the built-in software tools in IOS and ESXi.

Oddity: RITE Capture, Tunnels?
Interestingly, you can also use RITE to capture traffic to a RAM buffer and export it to a pcap file. I don't understand why you would use this instead of the much more flexible Embedded Packet Capture Feature, though.

Another thing I've wondered is whether you could use a L2 tunnel to send the mirrored traffic elsewhere in the network. The destination interface must be a physical Ethernet interface, but it would be interesting to try using a L2TPv3 tunnel from an Ethernet interface to another router--I have no idea if this would work.

Production Use?
Cisco makes the UCS E-series blades for ISR G2 routers that let you run a hypervisor on a blade inside your router chassis. These things have an external Ethernet port on them, so you should be able to connect a RITE export interface to the external port on an E-series blade, and run Security Onion inside your router. I've always wanted to try this, but I haven't been able to get funding yet to test it.

Friday, March 21, 2014

Quick Thoughts on the Micro Data Center

Here's something that's been on my radar lately: while all the talk in the networking world seems to be about the so-called "massively scalable" data center, almost all of the people I talk to in my world are dealing with the fact that data centers are rapidly getting smaller due to virtualization efficiencies. This seems to be the rule rather than the exception for small-to-medium sized enterprises.

In the micro data center that sits down the hall from me, for example, we've gone from 26 physical servers to 18 in the last few months, and we're scheduled to lose several more as older hypervisor hosts get replaced with newer, denser models. I suspect we'll eventually stabilize at around a dozen physical servers hosting in the low hundreds of VMs. We could get much denser, but things like political boundaries inevitably step in to keep the count higher than it might be otherwise. The case is similar in our other main facility.

From a networking perspective, this is interesting: I've heard vendor and VAR account managers remark lately that virtualization is cutting into their hardware sales. I'm most familiar with Cisco's offerings, and at least right now they don't seem to be looking at the micro-DC market as a niche: high-port count switches are basically all that are available. Cisco's design guide for the small-to-medium data center starts in the 100-300 10GE port range, which with modern virtualization hosts will support quite a few thousand typical enterprise VMs.

Having purchased the bulk of our older-generation servers before 10GE was cheap, we're just getting started with 10GE to the server access layer. Realistically, within a year or so a pair of redundant, reasonably feature-rich 24-32 port 10GE switches will be all we need for server access, probably in 10GBASE-T. Today, my best Cisco option seems to be the Nexus 9300 series, but it still has a lot of ports I'll never use.

One thought I've had is to standardize on the Catalyst 4500-X for all DC, campus core, and campus distribution use. With VSS, the topologies are simple. The space, power, and cooling requirements are minimal, and the redundancy is there. It has good layer 3 capabilities, along with excellent SPAN and NetFlow support. The only thing it seems to be lacking today is an upgrade path to 40GE, but that may be acceptable in low-port-density environments. Having one platform to manage would be nice. The drawbacks, of course, are a higher per-port cost and lack of scalability -- but again, scalability isn't really the problem.

Comments welcome.

Wednesday, March 5, 2014

Packet Capture in Diverse / Tunneled Networks?

(With the usual caveats that I am just a hick from Colorado, I don't know what I'm talking about, etc.)

I just read Pete Welcher's superb series on NSX, DFA, ACI, and other SDN stuff on the Chesapeake Netcraftsmen blog, and it helped me think more clearly about a problem that's been bothering me for a long time: how do we do realistically scalable packet capture in networks that make extensive use of ECMP and/or tunnels? Here's a sample network that Pete used:






Conventionally, we place packet capture devices at choke points in the network. But in medium-to-large data center designs, one of the main goals is to eliminate choke points: if we assume this is a relatively small standard ECMP leaf-spine design, each of the leaf switches has four equal-cost routed paths through the spine switches, and each spine switch has at least as many downlinks as there are leaf switches. The hypervisors each have two physical paths to the leaf switches, and in a high-density virtualization design we probably don't have a very good idea of what VM resides on what hypervisor at any point in time.

Now, add to that the tunneling features present in hypervisor-centric network virtualization schemes: traffic between two VMs attached to different hypervisors is tunneled inside VXLAN, GRE, or STT packets, depending on how you have things set up. The source and destination IP addresses of the "outer" hypervisor-to-hypervisor packet are not the sample as the "inner" VM-to-VM addresses, and presumably it's the latter that interest us. Thus, it's hard to even figure out which packets to capture. If we capture all of them (hard at any kind of scale; good 10G line-rate capture is still expensive and troublesome), we still have to filter for the inner tunnel headers to figure out what we're looking at.

What can we do? I see a few options:


  1. Put a huge mirror/tap switch between the leaf and spine. Gigamon makes some big ones, with up to 64 x 40GE ports or 256 x 10GE ports. When you max those out, start putting them at the end of each row. They advertise the ability to pop all kinds of different tunnel headers in hardware, along with lots of cool filtering and load-balancing capabilities.
  2. Buy or roll-your-own rack-mount packet capture appliances on commodity hardware, and run ad hoc SPAN sessions to them from the leaf switches.
  3. Install hypervisor-based packet capture VMs on all your hypervisors and capture from promiscuous mode vSwitches. There are lots of commercial solutions here, or you could roll your own. Update: Pete Welcher responded on Twitter and mentioned the option of doing packet capture pre-tunnel-encap or post-tunnel-decap. That's originally what I was thinking of with this option, but after reviewing a couple of his posts again, it appears there may be scenarios where the hypervisor makes tunnels to itself, so a better way of doing it might be to implement a packet capture API in the hypervisor itself that can control the point in the tunnel chain where the capture takes place. The next question is: where do we retrieve the capture? Does the API send the capture to a VM, save it on a datastore, dump it to a physical port, send it via another tunnel akin to ERSPAN? I'd want multiple options.
  4. Make sure Wireshark, tshark, or tcpdump is installed on every VM.
  5. Give up on intra-DC capture and focus only at the ingress/egress points.
Option 1 is the only one that really confronts both the network diversity and tunnel encapsulation head-on. Those boxes and their administrative overhead don't come cheap, but today this is probably the most fiddle-free option. The other options require a lot of customization and manual intervention that may or may not interfere with change-control procedures, and don't provide obvious solutions for de-obfuscating the tunneled traffic. Option 2 also suffers from serious scalability problems in ECMP designs. Option 5 just avoids the problem, but might work for some people.

However, the whole point of these designs is "SDN". What I *hope* is going to happen as SDN controllers start to become available is that the controller will be sufficiently aware of VM location that it can instruct the appropriate vSwitch OR leaf-switch OR spine switch to copy packets that meet a certain set of criteria to a particular destination port. Call it super-SPAN (can you tell I'm not headed for a new career in product naming?). It would be nice to be able to define the copied packets in different ways:
  • Conventional L3/L4 5-tuple. This would be nice because it could be informed by NetFlow/IPFIX data, without the need for DPI on the flow-exporter.
  • VM DNS name, port profile, or parent hypervisor.
  • QoS class.
  • "Application profile" -- it remains to be seen exactly what this means, but this is one of those SDN holy-grail things that allows more granular definition of traffic types.
Finally, it would be nice if the controller was smart enough to be able to load-balance the copied packets when necessary, so that the same capture target sees both sides of a given flow.

Again, I know nothing about plans for this stuff from any vendor. But I hope the powers-that-be in the SDN/etc world are thinking about at least some of these kinds of capabilities.

And... about the time they get that all figured out, we'll have to be dealing with a bunch of that traffic being encrypted between VMs or hypervisors...

Friday, February 14, 2014

Programmatically Configuring Interface Descriptions from Phone Descriptions

I wrote some Python code that allows you to do the following:
  1. Query a Catalyst switch CDP neighbor table from its HTTPS interface,
  2. Extract the device names of the attached IP phones,
  3. Query Communications Manager for the IP phone device description, 
  4. Apply the device description as the switch interface description.
Obviously, this makes it much easier to see whose phone is attached to a switch port.

I hope that this example saves someone the head-banging that I incurred while trying to figure out the AXL XML/SOAP API for Communications Manager.

I haven't tested this extensively; all my testing has been on Catalyst 3560 and 3750 switches and CUCM version 8.6. Using the --auto switch to automatically configure the switch is quite slow; this is a limitation of the HTTPS interface rather than the script code. It may be faster to leave that option off and manually copy/paste the printed configuration if you're in a hurry.

Note that your switch must be configured to allow configuration via the HTTPS interface; you may need to modify your TACACS/etc. configurations accordingly.

All the relevant info is in the Github repo.