tag:blogger.com,1999:blog-50296899811581135882024-03-19T02:37:17.208-06:00Loopback MountainVery Intermittent Geekery: Cisco IOS, VoIP, Infosec, etc.Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.comBlogger100125tag:blogger.com,1999:blog-5029689981158113588.post-67532067557505946982015-09-18T09:02:00.002-06:002015-09-19T12:16:01.685-06:00Moving to GitHub PagesI've moved this blog to <a href="https://jayswan.github.io/" target="_blank">GitHub Pages</a>.<br />
<br />
<a href="https://jayswan.github.io/rss.xml" target="_blank">The RSS feed is here. </a>Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com123tag:blogger.com,1999:blog-5029689981158113588.post-16313396733309993772015-07-15T13:12:00.000-06:002015-07-15T13:12:08.406-06:00Installing netmiko on Windows<a href="https://github.com/ktbyers/netmiko" target="_blank">Netmiko </a>is a Python module by <a href="https://twitter.com/kirkbyers" target="_blank">Kirk Byers</a> that provides a wrapper around the <a href="http://www.paramiko.org/" target="_blank">Paramiko </a>SSH module for doing screen scraping and CLI automation on network devices.<br />
<br />
Paramiko has some dependencies that make installation on Windows a tad tricky. Here's a quick way to get it done:<br />
<br />
<ol>
<li>Install <a href="https://store.continuum.io/cshop/anaconda/" target="_blank">Anaconda</a>.</li>
<li>From the Anaconda shell, run "conda install paramiko".</li>
<li>From the Anaconda shell, run "pip install scp".</li>
<li>Install <a href="https://www.git-scm.com/downloads" target="_blank">git for Windows</a>.</li>
<li>Clone netmiko with "git clone <a href="https://github.com/ktbyers/netmiko">https://github.com/ktbyers/netmiko</a>"</li>
<li>cd into the netmiko directory and run "python setup.py install".</li>
</ol>
Done! Screen scrape away, and don't forget to hound your vendors for real APIs... :-)Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com79tag:blogger.com,1999:blog-5029689981158113588.post-41537302606067099962015-07-14T11:35:00.000-06:002015-07-14T11:37:41.378-06:00Extracting Traffic from Rolling Capture FilesEvery so often I need to extract a subset of traffic from a set of rolling timestamped pcap files. One common place I do this is with <a href="https://github.com/Security-Onion-Solutions/security-onion" target="_blank">Security Onion</a>; one of the great features of SO is its full-packet-capture feature: you can easily pivot from Snort, Suricata, or Bro logs to a full packet capture view, or download the associated pcap file.<br />
<br />
But what if you don't have an associated alert or Bro log entry? Or if you're doing pcap on some system that's not as user-friendly as Security Onion, but nonetheless supports rolling captures?<br />
<br />
The way I usually do this is with <span style="font-family: "Courier New",Courier,monospace;">find </span>and <span style="font-family: "Courier New",Courier,monospace;">xargs</span>. Here's an example of my most common workflow, using timestamps as the filtering criteria for <span style="font-family: "Courier New",Courier,monospace;">find</span>:<br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-size: x-small;">> find . -newerct "16:07" ! -newerct "16:10" | xargs -I {} tcpdump -r {} -w /tmp/{} host 8.8.8.8</span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-size: x-small;">> cd /tmp</span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-size: x-small;">> mergecap -w merged.pcap *.pcap</span></span><br />
<br />
Translated:<br />
<ol>
<li>Find all files in the current directory created after 16:07 but not created after 16:10. This requires GNU <span style="font-family: "Courier New",Courier,monospace;">find</span> 4.3.3 or later. It supports many different time and date formats.</li>
<li>Using <span style="font-family: "Courier New",Courier,monospace;">xargs</span>, filter each file with the "<span style="font-family: "Courier New",Courier,monospace;">host 8.8.8.8</span>" BPF expression and write it to <span style="font-family: "Courier New",Courier,monospace;">/tmp</span> with the same filename.</li>
<li>Merge all the .pcap files in <span style="font-family: "Courier New",Courier,monospace;">/tmp</span> into <span style="font-family: "Courier New",Courier,monospace;">merged.pcap</span>.</li>
</ol>
You can easily modify this workflow to fit other use cases. Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com9tag:blogger.com,1999:blog-5029689981158113588.post-65149577242102121742015-05-15T14:52:00.004-06:002015-05-15T14:52:58.041-06:00More ADN (Awk Defined Networking)Want to know how many IPv4 nodes are in each of your VLANs? Use ADN:<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">ssh myswitch 'sh arp | i Vlan' | awk '{print $NF}' | sort | uniq -c | sort -rn</span></span><br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;"><br /></span></span>
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-size: x-small;"> 79 Vlan38<br /> 65 Vlan42<br /> 58 Vlan34<br /> 22 Vlan36<br /> 21 Vlan32<br /> 20 Vlan40<br /> 9 Vlan3<br /> 7 Vlan8<br /> 5 Vlan6<br /> 5 Vlan204<br /> 5 Vlan203<br /> 5 Vlan2<br /> 4 Vlan74<br /> 3 Vlan82<br /> 3 Vlan4</span></span>Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com5tag:blogger.com,1999:blog-5029689981158113588.post-78450262177691663792015-04-24T10:39:00.000-06:002015-04-24T10:42:47.032-06:00ADN - Awk Defined NetworkingBecause I have yet to transition to a completely software-defined network in which everything configures itself (wink wink), I still have to do tasks like bulk VLAN changes.<br />
<br />
Thanks to a recent innovation called ADN, or "AWK Defined Networking", I can do this in a shorter time window that the average bathroom break. For example, I just had a request to change all ports on a large access switch stack that are currently in VLAN 76 to VLAN 64:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"># <b>ssh switch_name.foo.com 'show int status | i _76_' | grep Gi | awk '{print "int ",$1,"\n","description PC/Phone","\n","switchport access vlan 64"}'</b><br />Password: ***</span><br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">int Gi1/0/25<br /> description PC/Phone<br /> switchport access vlan 64<br />int Gi1/0/26<br /> description PC/Phone<br /> switchport access vlan 64</span><br />
<span style="font-family: "Courier New",Courier,monospace;">[many more deleted] </span><br />
<br />
Then I copied and pasted the results into config mode. Back to lounging on the beach.<br />
<br />
<a href="http://ferd.ca/awk-in-20-minutes.html" target="_blank">Not even any Python skills required!</a>Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com4tag:blogger.com,1999:blog-5029689981158113588.post-46458937525700426052015-03-26T20:17:00.001-06:002015-03-26T20:17:28.814-06:00Quick Example: Elasticsearch Bulk Index API with PythonA <a href="https://gist.github.com/jayswan/a8d9920ef74516a02fe1" target="_blank">quick example</a> that shows how to use <a href="http://elasticsearch-py.readthedocs.org/en/master/helpers.html#elasticsearch.helpers.bulk" target="_blank">Elasticsearch bulk indexing from the Python client</a>. This is dramatically faster than indexing documents one at a time in a loop with the index() method.<br />
<br />
<script src="https://gist.github.com/jayswan/a8d9920ef74516a02fe1.js"></script>
Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com10tag:blogger.com,1999:blog-5029689981158113588.post-52436883680826438022015-02-28T20:27:00.001-07:002015-02-28T20:32:32.149-07:00Filtering .raw fields with Python Elasticsearch DSL High-Level ClientIt took me a while to figure out how to search the <span style="font-family: "Courier New",Courier,monospace;">not_analyzed</span> ".raw" fields created by Logstash in Elasticsearch indices, using the high-level Python Elasticsearch client. Because keyword arguments can't have attributes, Python throws an error if you try it the intuitive way (this assumes you've already set up a client as <b><i>es</i></b> and an index as <b><i>i</i></b>, as shown in the <a href="http://elasticsearch-dsl.readthedocs.org/en/latest/index.html#" target="_blank">docs</a>):<br />
<br />
<script src="https://gist.github.com/jayswan/d4ddd71a35bb5f1ad86f.js"></script>
Instead, you create a dictionary with your parameters and unpack it using the ** operator:<br />
<br />
<script src="https://gist.github.com/jayswan/3a7621d909b15c832cfb.js"></script>
This produces the Elasticsearch query we want:<br />
<br />
<script src="https://gist.github.com/jayswan/c04eee5287cc7cbc5ea1.js"></script>Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com7tag:blogger.com,1999:blog-5029689981158113588.post-9208405159731178912015-01-15T15:17:00.000-07:002015-01-15T15:17:05.343-07:00Pleasing terminal colors on Security OnionTo get the lovely Solarized theme working in Security Onion:<br />
<ol>
<li><span style="font-family: "Courier New",Courier,monospace;">sudo apt-get install gnome-terminal</span></li>
<ol>
<li>I'm sure there's a way to get in working in the default xfce4 terminal, but I couldn't figure it out. </li>
</ol>
<li>Follow instructions here: <a href="http://stackoverflow.com/questions/23118916/configuring-solarized-colorscheme-in-gnome-terminal-tmux-and-vim">http://stackoverflow.com/questions/23118916/configuring-solarized-colorscheme-in-gnome-terminal-tmux-and-vim</a></li>
</ol>
Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com4tag:blogger.com,1999:blog-5029689981158113588.post-59448512895544653802015-01-08T15:21:00.000-07:002015-01-08T15:21:02.746-07:00Problems with kvm-ok in VIRL with VMWare PlayerI'm installing Cisco VIRL, and despite following the instructions regarding nested virtualization settings, the kvm-ok command was still complaining. I needed to edit the .vmx file for the VIRL VM and add/edit the following:<br />
<br />
<pre><code>monitor.virtual_mmu = "hardware"
monitor.virtual_exec = "hardware"
vhv.enable = "TRUE"
monitor_control.restrict_backdoor = "true"</code></pre>
<pre><code> </code></pre>
<pre><code> </code></pre>
Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com3tag:blogger.com,1999:blog-5029689981158113588.post-37279981067909462802015-01-02T10:45:00.001-07:002015-01-02T10:45:42.416-07:00My Network ToolkitA while back, Chris Marget of <a href="http://www.fragmentationneeded.net/" target="_blank">Fragmentation Needed</a> posted a run-down of his comprehensive and extremely clever <a href="http://www.fragmentationneeded.net/2013/07/network-toolkit.html" target="_blank">network toolkit</a>. Because I'm something of a weight weenie, mine is a lot more slimmed down. I thought I'd post it here:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9NtwneI7n6IlWwnVuSSv3n-E0WBU6Vo6AH7dpm793kYasObrklPlCezOCs5lV6e-6Ch24scPi8nZs534RxBO9Bng0tyvm3s7mEDDS36EFqNKULPvoS38cfKRIGir6Dei-VTeawv7IHEU/s1600/toolkit.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9NtwneI7n6IlWwnVuSSv3n-E0WBU6Vo6AH7dpm793kYasObrklPlCezOCs5lV6e-6Ch24scPi8nZs534RxBO9Bng0tyvm3s7mEDDS36EFqNKULPvoS38cfKRIGir6Dei-VTeawv7IHEU/s1600/toolkit.jpg" height="239" width="320" /></a></div>
<br />
<br />
<br />
<br />
The contents:<br />
<br />
<ol>
<li>Two random USB drives (in case I need to leave one with somebody).</li>
<li>Single-mode and multi-mode LC fiber loopback plugs.</li>
<li>Rack PDU plug adapter.</li>
<li>Awesome PicQuic compact screwdriver (thanks to Chris's post).</li>
<li>T1 loopback plug (red) (because we still have T1s out here in the boonies).</li>
<li>Cat-6 pass-through plug (white).</li>
<li>Crossover adapter (orange).</li>
<li>Sharpie.</li>
<li>Console setup:</li>
<ol>
<li>USB-to-DB9 adapter.</li>
<li>DB9-to-RJ45 adapter.</li>
<li>Flat Cat-6 cable.</li>
<li>Rollover adapter.</li>
<li>Velcro tie</li>
<li>Flat Cat-6 cable with velcro tie.</li>
</ol>
</ol>
The console setup could probably be improved by adding a DB9 null-modem adapter. The coolest thing (IMO) that I'm missing from Chris's setup is the Bluetooth console adapter -- maybe one day. <br />
<br />
The Fenix AA light and Leatherman Skeletool CX almost always live in a pocket rather than the kit and go with me everywhere. The kit all fits into a small zippered case that used to hold a Dell laptop power supply.<br />
<br />
My main goal here was to have all the hard-to-find professional stuff in one small package. I have a separate "personal" kit that contains stuff like headphones, USB cables, and chargers for personal electronics.Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com2tag:blogger.com,1999:blog-5029689981158113588.post-34951906303775479332014-12-05T08:49:00.001-07:002014-12-05T08:49:28.982-07:00Imposing Artificial Limitations to Develop SkillsI'm a big fan of imposing artificial limitations on yourself in order to aid skill development. Here are some quick ideas:<br />
<br />
<ul>
<li>When troubleshooting network devices from the CLI, try not to look at the configuration. Use only "show" or "debug" commands instead. I found this enormously beneficial when practicing for CCIE.</li>
<li>When troubleshooting larger operational issues or learning a new environment, try not to log into individual devices at all. Force yourself to use only your network management system, NetFlow, packet captures, or host-based tools like ping, traceroute, or nmap.</li>
<li>When learning automation or orchestration skills, force yourself to write scripts, run API calls, or use your favorite orchestration tool to do simple things, even if it doesn't seem like they merit the extra effort.</li>
</ul>
Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com2tag:blogger.com,1999:blog-5029689981158113588.post-64465236741032726062014-07-01T19:24:00.001-06:002014-07-02T09:06:21.573-06:00Simple Python Syslog CounterRecently I did a <a href="http://packetpushers.net/show-192-logging-design-best-practices/" target="_blank">Packet Pushers episode about log management</a>. In it, I mentioned some of the custom Python scripts that I run to do basic syslog analysis, and someone asked about them in the comments.<br />
<br />
The script I'm presenting here isn't one of the actual ones that I run in production, but it's close. The real one sends emails, does DNS lookups, keeps a "rare messages" database using sqlite3, and a few other things, but I wanted to keep this simple.<br />
<br />
One of the problems I see with getting started with log analysis is that people tend to approach it like a typical vendor RFP project: list some requirements, survey the market, evaluate and buy a product to fit your requirements. Sounds good, right? The problem with log analysis is that often you don't know what your requirements really are until you start looking at data.<br />
<br />
A simple message counting script like this lets you look at your data, and provides a simple platform on which you can start to iterate to find your specific needs. It also lets us look at some cool Python features.<br />
<br />
I don't recommend pushing this too far: once you have a decent idea of what your data looks like and what you want to do with it, set up <a href="http://logstash.net/" target="_blank">Logstash</a>, <a href="http://graylog2.com/" target="_blank">Graylog2</a>, or a similar commercial product like Splunk (if you can afford it).<br />
<br />
That said, <a href="https://gist.github.com/jayswan/96df3f0b9606f2ce84f2" target="_blank">here's the Python</a>:<br />
<br />
<br />
<script src="https://gist.github.com/jayswan/96df3f0b9606f2ce84f2.js"></script>
I tried to make this as self-documenting as possible. You run it from the CLI with a syslog file as the argument, and you get this:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ python simple_syslog_count.py sample.txt<br /> 214 SEC-6-IPACCESSLOGP<br /> 15 SEC-6-IPACCESSLOGRL<br /> 10 LINEPROTO-5-UPDOWN<br /> 10 LINK-3-UPDOWN<br /> 7 USER-3-SYSTEM_MSG<br /> 4 STACKMGR-4-STACK_LINK_CHANGE<br /> 4 DUAL-5-NBRCHANGE<br /> 3 IPPHONE-6-UNREGISTER_NORMAL<br /> 3 CRYPTO-4-PKT_REPLAY_ERR<br /> 3 SEC-6-IPACCESSLOGRP<br /> 3 SEC-6-IPACCESSLOGSP<br /> 2 SSH-5-SSH2_USERAUTH<br /> 2 SSH-5-SSH2_SESSION<br /> 2 SSH-5-SSH2_CLOSE<br /><br />10.1.16.12<br /><br /> 6 SEC-6-IPACCESSLOGP<br /><br />10.1.24.3<br /><br /> 2 LINEPROTO-5-UPDOWN<br /> 2 LINK-3-UPDOWN</span><br />
<br />
[Stuff deleted for brevity]<br />
<br />
For Pythonistas, the script makes use of a few cool language features:<br />
<ul>
</ul>
<h3>
Named, Compiled rRgexes</h3>
<ul>
<li>We can name a regex match with the<span style="font-family: "Courier New",Courier,monospace;"> </span><span class="s"><span style="font-family: "Courier New",Courier,monospace;">(?P<yourname>PATTERN)</yourname></span> syntax, which makes it easy to understand it when it's referenced later with the <span style="font-family: "Courier New",Courier,monospace;">.group('<yourname>') </yourname></span>method on the match object.</span></li>
<li><span class="s">This is demonstrated in lines 36-39 and 58-59 of the gist shown above. </span></li>
<li><span class="s">It would be more efficient to capture these fields by splitting the line with the .split() string method, but I wanted the script to work for unknown field positions -- hence the regex. </span> </li>
</ul>
<h3>
Multiplication of Strings</h3>
<ul>
<li>We control indentation by multiplying the ' ' string (that a single space enclosed in quotes) by an integer value in the print_counter function (line 50).</li>
<ul>
<li>The reason this works is that the Python <span style="font-family: "Courier New",Courier,monospace;">str</span> class defines a special <span style="font-family: "Courier New",Courier,monospace;">__mul__ </span>method that controls how the * operator works for objects of that class:<br /><span style="font-family: "Courier New",Courier,monospace;">>>> 'foo'.__mul__(3)<br />'foofoofoo'<br />>>> 'foo' * 3<br />'foofoofoo'</span></li>
</ul>
</ul>
<h3>
collections.Counter Objects</h3>
<ul>
<li>Counter objects are a subclass of dictionaries that know how to count things. Jeremy Schulman talked about these in a comment on the <a href="http://unroutable.blogspot.com/2014/06/python-sets-handy-for-network-data.html" target="_blank">previous post</a>. Here, we use Counters to build both the overall message counts and the per-device message counts:</li>
</ul>
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">>>> my_msg = 'timestamp ip_address stuff %MY-4-MESSAGE:other stuff'<br />>>> CISCO_MSG = re.compile('%(?P<msg>.*?):')<br />>>> from collections import Counter<br />>>> test_counter = Counter()<br />>>> this_msg = re.search(CISCO_MSG,my_msg).group('msg')<br />>>> this_msg<br />'MY-4-MESSAGE'<br />>>> test_counter[this_msg] += 1<br />>>> test_counter<br />Counter({'MY-4-MESSAGE': 1}) </msg></span></blockquote>
<ul>
</ul>
<h3>
collections.defaultdict Dictionaries</h3>
<ul>
<li>It could get annoying when you're assigning dictionary values inside a loop, because you get errors when the key doesn't exist yet. This is a contrived example, but it illustrates the point:<br /><br /><span style="font-family: "Courier New",Courier,monospace;">>>> reporters = {}<br />>>> for reporter in ['1.1.1.1','2.2.2.2']:<br />... reporters[reporter].append['foo']<br />... <br />Traceback (most recent call last):<br /> File "<stdin>", line 2, in <module><br />KeyError: '1.1.1.1'</module></stdin></span><br /> </li>
<li>To fix this, you can catch the exception:<br /><br /><span style="font-family: "Courier New",Courier,monospace;">>>> reporters = {}<br />>>> for reporter in ['1.1.1.1','2.2.2.2']:<br />... try:<br />... reporters[reporter].append['foo']<br />... reporters[reporter].append['bar']<br />... except KeyError:<br />... reporters[reporter] = ['foo']<br />... reporters[reporter].append('bar')</span></li>
</ul>
<ul>
<li>As usual, though, Python has a more elegant way in the <span style="font-family: "Courier New",Courier,monospace;">collections</span> module: <span style="font-family: "Courier New",Courier,monospace;">defaultdict</span> </li>
</ul>
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">>>> from collections import defaultdict<br />>>> reporters = defaultdict(list)<br />>>> for reporter in ['1.1.1.1','2.2.2.2']:<br />... reporters[reporter].append('foo')<br />... reporters[reporter].append('bar')<br />>>> reporters<br />defaultdict(<type list="">, {'1.1.1.1': ['foo', 'bar'], '2.2.2.2': ['foo', 'bar']})</type></span></blockquote>
In the syslog counter script, we use a <span style="font-family: "Courier New",Courier,monospace;">collections.Counter</span> object as the type for our defaultdict. This allows us to build a per-syslog-reporter dictionary that shows how many times each message appears for each reporter, while only looping through the input once (line 66):<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> per_reporter_counts[reporter][msg] += 1</span><br />
<br />
Here, the dictionary <span style="font-family: "Courier New",Courier,monospace;">per_reporter_counts</span> has the IPv4 addresses of the syslog reporters as keys, with a <span style="font-family: "Courier New",Courier,monospace;">Counter</span> object as the value holding the counts for each message type:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">>>> from collections import Counter,defaultdict<br />>>> per_reporter_counts = defaultdict(Counter)<br />>>> per_reporter_counts['1.1.1.1']['SOME-5-MESSAGE'] += 1<br />>>> per_reporter_counts<br />defaultdict(<class collections.counter="">, {'1.1.1.1': Counter({'SOME-5-MESSAGE': 1})})<br />>>> per_reporter_counts['1.1.1.1']['SOME-5-MESSAGE'] += 5<br />>>> per_reporter_counts<br />defaultdict(<class collections.counter="">, {'1.1.1.1': Counter({'SOME-5-MESSAGE': 6})})</class></class></span><br />
<br />
If you got this far, you can go implement it for IPv6 addresses. :-)Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com5tag:blogger.com,1999:blog-5029689981158113588.post-57515171426992651962014-06-20T10:55:00.001-06:002014-06-20T10:55:02.147-06:00Python Sets: Handy for Network DataMy Python-related posts seem to get the most reads, so here's another one!<br />
<br />
A problem that comes up fairly often in networking is finding the number of occurrences of unique items in a large collection of data: let's say you want to find all of the unique IP addresses that accessed a website, traversed a firewall, got denied by an ACL, or whatever. Maybe you've extracted the following list from a log file:<br />
<br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">1.1.1.1</span></span><br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">2.2.2.2</span></span><br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">3.3.3.3</span></span><br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">1.1.1.1</span></span><br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">5.5.5.5</span></span><br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">5.5.5.5</span></span><br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">1.1.1.1</span></span><br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">2.2.2.2 </span></span><br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">...</span></span><br />
<br />
and you need to reduce this to:<br />
<br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">1.1.1.1</span></span><br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">2.2.2.2</span></span><br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">3.3.3.3</span></span><br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">5.5.5.5</span></span><br />
<br />
In other words, we're removing the duplicates. In low-level programming languages, removing duplicates is a bit of a pain: generally you need to implement an efficient way to sort an array of items, then traverse the sorted array to check for adjacent duplicates and remove them. In a language that has dictionaries (also known as hash tables or associative arrays), you can do it by adding each item as a key in your dictionary with an empty value, then extract the keys. In Python:<br />
<br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">>>> items = ['1.1.1.1','2.2.2.2','3.3.3.3','1.1.1.1','5.5.5.5','5.5.5.5','1.1.1.1','2.2.2.2']<br />>>> d = {}<br />>>> for item in items:<br />... d[item] = None<br />...<br />>>> d<br />{'5.5.5.5': None, '3.3.3.3': None, '1.1.1.1': None, '2.2.2.2': None}<br />>>> unique = d.keys()<br />>>> unique<br />['5.5.5.5', '3.3.3.3', '1.1.1.1', '2.2.2.2']</span></span><br />
<br />
or, more concisely using a dictionary comprehension:<br />
<br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">>>> {item:None for item in items}.keys()<br />['5.5.5.5', '3.3.3.3', '1.1.1.1', '2.2.2.2']</span></span><br />
<br />
Python has an even better way, however: the "set" type, which emulates the mathematical idea of a set as a collection of distinct items. If you create an empty set and add items to it, duplicates will automatically be thrown away:<br />
<br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">>>> s = set()<br />>>> s.add('1.1.1.1')<br />>>> s<br />set(['1.1.1.1'])<br />>>> s.add('2.2.2.2')<br />>>> s.add('1.1.1.1')<br />>>> s<br />set(['1.1.1.1', '2.2.2.2'])<br />>>> for item in items:<br />... s.add(item)<br />...<br />>>> s<br />set(['5.5.5.5', '3.3.3.3', '1.1.1.1', '2.2.2.2'])</span></span><br />
<br />
Predictably, you can use set comprehensions just like list comprehensions to do the same thing as a one liner:<br />
<br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">>>> {item for item in items}<br />set(['5.5.5.5', '3.3.3.3', '1.1.1.1', '2.2.2.2'])</span></span><br />
<br />
Or, if you have a list built already you can just convert it to a set:<br />
<br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">>>> set(items)<br />set(['5.5.5.5', '3.3.3.3', '1.1.1.1', '2.2.2.2'])</span></span><br />
<br />
Python also provides methods for the most common types of set operations: union, intersection, difference and symmetric difference. Because these methods accept lists or other iterables, you can quickly find similarities between collections of items:<br />
<br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">>>> items<br />['1.1.1.1', '2.2.2.2', '3.3.3.3', '1.1.1.1', '5.5.5.5', '5.5.5.5', '1.1.1.1', '2.2.2.2']<br />>>> more_items = ['1.1.1.1','8.8.8.8','1.1.1.1','7.7.7.7','2.2.2.2']<br />>>> set(items).intersection(more_items)<br />set(['1.1.1.1', '2.2.2.2'])</span></span><br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">>>> set(items).difference(more_items)<br />set(['5.5.5.5', '3.3.3.3'])</span></span><br />
<br />
Have fun!<br />
<br />Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com55tag:blogger.com,1999:blog-5029689981158113588.post-11130815433983937962014-04-02T09:54:00.001-06:002014-04-02T09:54:03.981-06:00Fun with Router IP Traffic Export and NSM<i><span style="font-size: large;">The Basics </span></i><br />
I finally got around to setting up <a href="http://blog.securityonion.net/" target="_blank">Security Onion</a> (the best network security monitoring package available) to monitor my home network, only to discover that my Cisco 891 router doesn't support support the right form of SPAN. Here's how I worked around it. The topology looks like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4qMLtBtiJ4_txUkskCn8FfnGGjZOdolPsDuIHv32E_4gf593z8b8C1_OlLCpzN5iLAyaGUy2e_D2VyMdhgKINanMJwthxZcsSCvXAbDLgnZRit0lU3H9uyTx0KW8fAKqFNjG1q6hKqu0/s1600/lab.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4qMLtBtiJ4_txUkskCn8FfnGGjZOdolPsDuIHv32E_4gf593z8b8C1_OlLCpzN5iLAyaGUy2e_D2VyMdhgKINanMJwthxZcsSCvXAbDLgnZRit0lU3H9uyTx0KW8fAKqFNjG1q6hKqu0/s1600/lab.png" height="328" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
The 891 router has an integrated 8-port switch module, so the simple case would have been a traditional SPAN setup; something like this:<br />
<span style="font-size: small;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace; font-size: small;"><span style="font-family: "Courier New",Courier,monospace;">! vlan 10 is the user VLAN</span> </span><br />
<span style="font-family: "Courier New",Courier,monospace; font-size: small;">monitor session 1 source interface vlan 10 <!------></span><br />
<span style="font-family: "Courier New",Courier,monospace; font-size: small;">monitor session 1 destination interface FastEthernet0</span><br />
<br />
with the server's monitoring NIC connected to FastEthernet0.<br />
<br />
The problem is that the 891 doesn't support using a VLAN as a source interface, and because of the way the embedded WAP works, a physical source interface won't work either. Hence, I turned to an obscure feature that's helped me occasionally in the past: <a href="http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_usr_cfg/configuration/15-mt/sec-usr-cfg-15-mt-book/sec-ip-traff-export.html" target="_blank">Router IP Traffic Export</a>. This is a feature for IOS software platforms that enables you to enable SPAN-like functions for almost any source interface.<br />
<br />
The configuration looks like this:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">ip traffic-export profile RITE_MIRROR<br /> interface FastEthernet0<br /> bidirectional<br /> mac-address 6805.ca21.2ddd</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">interface Vlan10</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> ip traffic-export apply RITE_MIRROR</span><br />
<br />
This takes all traffic routed across the Vlan10 SVI and sends it out the FastEthernet0 interface, rewriting the destination MAC address to the specified value. I used the MAC address of my monitoring NIC, but it shouldn't matter in this case because the monitoring NIC is directly attached. If I wanted to copy the traffic across a switched interface, it would matter.<br />
<br />
My ESXi host (a low-cost machine from <a href="https://zareason.com/" target="_blank">Zareason</a> with 32GB of RAM) has two physical NICs; one for all of the regular VM traffic (using 802.1q to separate VLANs if needed) and one for monitoring. The monitoring pNIC is attached to a promiscuous mode vSwitch in ESXi, which in turn is connected to the monitoring vNIC on the Security Onion VM. The effect of this is identical to SPAN-ing all the traffic from VLAN 10 to my Security Onion monitoring system; I get Snort, Bro, Argus, and full packet capture with just the built-in software tools in IOS and ESXi.<br />
<br /><span style="font-size: large;"><i>Oddity: RITE Capture, Tunnels?</i></span><br />
Interestingly, you can also use RITE to <a href="http://www.cisco.com/c/en/us/td/docs/ios/12_4t/12_4t11/ht_rawip.html#wp1051438" target="_blank">capture traffic to a RAM buffer and export it to a pcap file</a>. I don't understand why you would use this instead of the much more flexible <a href="http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/epc/configuration/15-mt/epc-15-mt-book/nm-packet-capture.html" target="_blank">Embedded Packet Capture Feature</a>, though.<br />
<br />
Another thing I've wondered is whether you could use a L2 tunnel to send the mirrored traffic elsewhere in the network. The destination interface must be a physical Ethernet interface, but it would be interesting to try using a L2TPv3 tunnel from an Ethernet interface to another router--I have no idea if this would work. <br />
<br />
<i><span style="font-size: large;">Production Use?</span></i><br />
Cisco makes the <a href="http://www.cisco.com/c/dam/en/us/products/collateral/servers-unified-computing/ucs-e-series-servers/data_sheet_c78-705787.pdf" target="_blank">UCS E-series blades</a> for ISR G2 routers that let you run a hypervisor on a blade inside your router chassis. These things have an external Ethernet port on them, so you should be able to connect a RITE export interface to the external port on an E-series blade, and run Security Onion inside your router. I've always wanted to try this, but I haven't been able to get funding yet to test it.Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com3tag:blogger.com,1999:blog-5029689981158113588.post-6282890813902666032014-03-21T16:04:00.000-06:002014-03-21T16:21:16.412-06:00Quick Thoughts on the Micro Data CenterHere's something that's been on my radar lately: while all the talk in the networking world seems to be about the so-called "massively scalable" data center, almost all of the people I talk to in my world are dealing with the fact that data centers are rapidly getting smaller due to virtualization efficiencies. This seems to be the rule rather than the exception for small-to-medium sized enterprises.<br />
<br />
In the micro data center that sits down the hall from me, for example, we've gone from 26 physical servers to 18 in the last few months, and we're scheduled to lose several more as older hypervisor hosts get replaced with newer, denser models. I suspect we'll eventually stabilize at around a dozen physical servers hosting in the low hundreds of VMs. We could get much denser, but things like political boundaries inevitably step in to keep the count higher than it might be otherwise. The case is similar in our other main facility.<br />
<br />
From a networking perspective, this is interesting: I've heard vendor and VAR account managers remark lately that virtualization is cutting into their hardware sales. I'm most familiar with Cisco's offerings, and at least right now they don't seem to be looking at the micro-DC market as a niche: high-port count switches are basically all that are available. Cisco's design guide for the small-to-medium data center starts in the 100-300 10GE port range, which with modern virtualization hosts will support quite a few thousand typical enterprise VMs.<br />
<br />
Having purchased the bulk of our older-generation servers before 10GE was cheap, we're just getting started with 10GE to the server access layer. Realistically, within a year or so a pair of redundant, reasonably feature-rich 24-32 port 10GE switches will be all we need for server access, probably in 10GBASE-T. Today, my best Cisco option seems to be the Nexus 9300 series, but it still has a lot of ports I'll never use.<br />
<br />
One thought I've had is to standardize on the Catalyst 4500-X for all DC, campus core, and campus distribution use. With VSS, the topologies are simple. The space, power, and cooling requirements are minimal, and the redundancy is there. It has good layer 3 capabilities, along with excellent SPAN and NetFlow support. The only thing it seems to be lacking today is an upgrade path to 40GE, but that may be acceptable in low-port-density environments. Having one platform to manage would be nice. The drawbacks, of course, are a higher per-port cost and lack of scalability -- but again, scalability isn't really the problem.<br />
<br />
Comments welcome.Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com5tag:blogger.com,1999:blog-5029689981158113588.post-45954677469018973152014-03-05T20:26:00.001-07:002014-03-09T09:04:28.814-06:00Packet Capture in Diverse / Tunneled Networks?(With the usual caveats that I am just a hick from Colorado, I don't know what I'm talking about, etc.)<br />
<br />
I just read <a href="http://netcraftsmen.net/component/easyblog/blogger/listings/pwelcher.html?Itemid=374" target="_blank">Pete Welcher's superb series</a> on NSX, DFA, ACI, and other SDN stuff on the <a href="http://netcraftsmen.net/blogs.html" target="_blank">Chesapeake Netcraftsmen</a> blog, and it helped me think more clearly about a problem that's been bothering me for a long time: how do we do realistically scalable packet capture in networks that make extensive use of ECMP and/or tunnels? Here's a sample network that Pete used:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://netcraftsmen.net/images/stories/71/20140103-figure1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://netcraftsmen.net/images/stories/71/20140103-figure1.jpg" height="211" width="320" /></a></div>
<br />
<br />
<br />
<br />
<br />
Conventionally, we place packet capture devices at choke points in the network. But in medium-to-large data center designs, one of the main goals is to eliminate choke points: if we assume this is a relatively small standard ECMP leaf-spine design, each of the leaf switches has four equal-cost routed paths through the spine switches, and each spine switch has at least as many downlinks as there are leaf switches. The hypervisors each have two physical paths to the leaf switches, and in a high-density virtualization design we probably don't have a very good idea of what VM resides on what hypervisor at any point in time.<br />
<br />
Now, add to that the tunneling features present in hypervisor-centric network virtualization schemes: traffic between two VMs attached to different hypervisors is tunneled inside VXLAN, GRE, or STT packets, depending on how you have things set up. The source and destination IP addresses of the "outer" hypervisor-to-hypervisor packet are not the sample as the "inner" VM-to-VM addresses, and presumably it's the latter that interest us. Thus, it's hard to even figure out which packets to capture. If we capture all of them (hard at any kind of scale; good 10G line-rate capture is still expensive and troublesome), we still have to filter for the inner tunnel headers to figure out what we're looking at.<br />
<br />
What can we do? I see a few options:<br />
<br />
<br />
<ol>
<li>Put a huge mirror/tap switch between the leaf and spine. Gigamon makes some big ones, with up to 64 x 40GE ports or 256 x 10GE ports. When you max those out, start putting them at the end of each row. They advertise the ability to pop all kinds of different tunnel headers in hardware, along with lots of cool filtering and load-balancing capabilities.</li>
<li>Buy or roll-your-own rack-mount packet capture appliances on commodity hardware, and run ad hoc SPAN sessions to them from the leaf switches.</li>
<li>Install hypervisor-based packet capture VMs on all your hypervisors and capture from promiscuous mode vSwitches. There are lots of commercial solutions here, or you could roll your own. Update: <a href="https://twitter.com/pjwelcher" target="_blank">Pete Welcher</a> responded on Twitter and mentioned the option of doing packet capture pre-tunnel-encap or post-tunnel-decap. That's originally what I was thinking of with this option, but after reviewing a couple of his posts again, it appears there may be scenarios where the hypervisor makes tunnels to itself, so a better way of doing it might be to implement a packet capture API in the hypervisor itself that can control the point in the tunnel chain where the capture takes place. The next question is: where do we retrieve the capture? Does the API send the capture to a VM, save it on a datastore, dump it to a physical port, send it via another tunnel akin to ERSPAN? I'd want multiple options.</li>
<li>Make sure Wireshark, tshark, or tcpdump is installed on every VM.</li>
<li>Give up on intra-DC capture and focus only at the ingress/egress points. </li>
</ol>
Option 1 is the only one that really confronts both the network diversity and tunnel encapsulation head-on. Those boxes and their administrative overhead don't come cheap, but today this is probably the most fiddle-free option. The other options require a lot of customization and manual intervention that may or may not interfere with change-control procedures, and don't provide obvious solutions for de-obfuscating the tunneled traffic. Option 2 also suffers from serious scalability problems in ECMP designs. Option 5 just avoids the problem, but might work for some people.<br />
<br />
However, the whole point of these designs is "SDN". What I *hope* is going to happen as SDN controllers start to become available is that the controller will be sufficiently aware of VM location that it can instruct the appropriate vSwitch OR leaf-switch OR spine switch to copy packets that meet a certain set of criteria to a particular destination port. Call it super-SPAN (can you tell I'm not headed for a new career in product naming?). It would be nice to be able to define the copied packets in different ways:<br />
<ul>
<li>Conventional L3/L4 5-tuple. This would be nice because it could be informed by NetFlow/IPFIX data, without the need for DPI on the flow-exporter.</li>
<li>VM DNS name, port profile, or parent hypervisor.</li>
<li>QoS class.</li>
<li>"Application profile" -- it remains to be seen exactly what this means, but this is one of those SDN holy-grail things that allows more granular definition of traffic types.</li>
</ul>
Finally, it would be nice if the controller was smart enough to be able to load-balance the copied packets when necessary, so that the same capture target sees both sides of a given flow.<br />
<br />
Again, I know nothing about plans for this stuff from any vendor. But I hope the powers-that-be in the SDN/etc world are thinking about at least some of these kinds of capabilities.<br />
<br />
And... about the time they get that all figured out, we'll have to be dealing with a bunch of that traffic being encrypted between VMs or hypervisors... Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com4tag:blogger.com,1999:blog-5029689981158113588.post-49356539814817175102014-02-14T15:05:00.003-07:002014-02-14T15:06:55.663-07:00Programmatically Configuring Interface Descriptions from Phone DescriptionsI wrote some Python code that allows you to do the following:<br />
<ol>
<li>Query a Catalyst switch CDP neighbor table from its HTTPS interface,</li>
<li>Extract the device names of the attached IP phones,</li>
<li>Query Communications Manager for the IP phone device description, </li>
<li>Apply the device description as the switch interface description.</li>
</ol>
Obviously, this makes it much easier to see whose phone is attached to a switch port.<br />
<br />
I hope that this example saves someone the head-banging that I incurred while trying to figure out the AXL XML/SOAP API for Communications Manager.<br />
<br />
I haven't tested this extensively; all my testing has been on Catalyst 3560 and 3750 switches and CUCM version 8.6. Using the --auto switch to automatically configure the switch is quite slow; this is a limitation of the HTTPS interface rather than the script code. It may be faster to leave that option off and manually copy/paste the printed configuration if you're in a hurry.<br />
<br />
Note that your switch must be configured to allow configuration via the HTTPS interface; you may need to modify your TACACS/etc. configurations accordingly.<br />
<br />
All the relevant info is in the <a href="https://github.com/jayswan/cdp_cucm" target="_blank">Github repo</a>.Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com2tag:blogger.com,1999:blog-5029689981158113588.post-3928502822272152972014-01-20T11:07:00.000-07:002014-01-20T19:15:44.732-07:00Why Network Engineers Should Learn ProgrammingBecause Microsoft Excel is not a text editor. Seriously.<br />
<br />
This is a followup to the previous post, inspired by Ethan Banks of <a href="http://packetpushers.net/" target="_blank">Packet Pushers</a>. If you do operational networking at all, you deal with text files all the time: logs, debug output, configuration files, command line diagnostics, and more. I'm constantly amazed when I see people open Word or Excel to do their text editing, often one keystroke at a time.<br />
<br />
The number one reason to learn basic programming is to automate that stuff. Personally, I use a combination of traditional Unix shell tools and Python to get the job done, but you could probably do it all with one or the other.<br />
<br />
There are lots of other reasons to learn programming too, many of which will be discussed on an upcoming Packet Pushers episode. But if you don't believe any of those, this one alone makes it worth the effort.<br />
<br />
Step away from the spreadsheet. Do it now.Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com1tag:blogger.com,1999:blog-5029689981158113588.post-83879445972205957082014-01-17T10:14:00.001-07:002014-01-17T10:14:17.720-07:00Quick Thoughts on Learning PythonI was scheduled to be a guest on an upcoming episode of the Packet Pushers podcast, on the topic of Python for network engineers. Unfortunately due to bad luck I'm not going to be able to make the recording. Here are some quick thoughts on learning Python. If you're already an expert programmer you already know how to learn languages, so this post isn't for you.<br />
<br />
Scenario 1: You've coded in another language, but you're not an expert.<br />
I would start with the basic Python class at <a href="https://developers.google.com/edu/python/?csw=1" target="_blank">Google Code</a>. It's targeted specifically at people who know basic programming skills in some other language. It was perfect for me; I went through the exercises and was able to quickly start writing simple, useful Python scripts.<br />
<br />
Scenario 2: You don't know how to write code at all.<br />
Start with the <a href="https://www.udacity.com/course/cs101" target="_blank">Udacity CS101</a> class if you like guided learning, or <a href="http://learnpythonthehardway.org/" target="_blank">Learn Python the Hard Way</a> if you prefer books. Be prepared to spend a lot of time on either. It's not easy the first time around.<br />
<br />
After you've gotten through one of those two scenarios, do the following:<br />
<br />
<br />
<ol>
<li>Spend time browsing the documentation for the Python Standard Library. Python is a large language, and chances are there's something in the standard library that will help you meet your goals. If you find yourself writing a lot of lines of code to accomplish something fairly simple, look harder. I recommend skimming the documentation for every module, then looking more carefully at the ones that interest you.</li>
<li><a href="http://www.amazon.com/Matt-Harrison/e/B0077BQLH6/ref=ntt_athr_dp_pel_pop_1" target="_blank">Matt Harrison's books</a> on basic and intermediate Python are excellent. I recommend buying them and reading them.</li>
<li>Jeff Knupp's <a href="http://www.jeffknupp.com/writing-idiomatic-python-ebook/" target="_blank">Writing Idiomatic Python</a> is really good as you gain skills. It's a bit rough around the edges, but it will help you avoid common beginner mistakes and is well worth the read.</li>
<li>Practice a lot. I enjoy the math puzzles on <a href="http://projecteuler.net/" target="_blank">Project Euler</a>. These are not Python specific, but their structure makes them well suited to quick problem solving in any language. Beware -- it's very addictive!</li>
</ol>
Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com2tag:blogger.com,1999:blog-5029689981158113588.post-55280261075930943392014-01-08T10:57:00.001-07:002014-01-08T10:57:29.781-07:00Using Bro DNS Logging for Network ManagementI was recently asked if someone in our desktop support group could get alerted when certain laptops connected to the corporate network. We have a lot of employees who work at industrial locations and rarely connect their machines to our internal networks, so the support group likes to take those rare opportunities to do management tasks that aren't otherwise automated.<br />
<br />
The two mechanisms that came to mind for alerting on these events are DHCP address assignment, and DNS autoregistration. While we do send DHCP logs to a central archive, the process of alerting on a frequently changing list of hostnames would be somewhat cumbersome. I have been looking for ways to use Bro for network management tasks, so this seemed like a natural use case.<br />
<br />
We already had Bro instances monitoring DNS traffic for two of our central DNS servers. I don't fully understand how Windows DNS autoregistration works, but from looking at the Bro logs, it appears that the DHCP server sends a DNS SOA query to the central DNS servers containing the hostname of the device to which it assigns a lease.<br />
<br />
I wanted to learn how to use the input framework in Bro 2.2, so I wrote the following script and loaded it from local.bro:<br />
<br />
<a href="https://gist.github.com/jayswan/8321141" target="_blank">https://gist.github.com/jayswan/8321141</a><br />
<script src="https://gist.github.com/jayswan/8321141.js"></script>
<br />
This raises a Bro notice whenever one of the hostnames in the hostnames.txt file is seen in a DNS SOA query. I then set up local.bro and broctl to email this notice type to the correct person.<br />
<br />
This works for now, but I'd love to hear from any more experienced Bro programmers about better ways to do it. Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com0tag:blogger.com,1999:blog-5029689981158113588.post-25364743526141654792014-01-02T16:58:00.002-07:002014-01-02T16:58:54.397-07:00New Year Resolution: Code CleanupI enjoyed Ethan Banks' post on <a href="http://ethancbanks.com/2013/12/29/new-years-thoughts-start-with-documentation/" target="_blank">New Year's Thoughts: Start with Documentation</a>, so I thought I'd write about what I'm doing this week: code cleanup. Over the last couple of years I've written a decent amount of code to automate mundane network management tasks. As quick one-off hacks have turned into things that I actually depend on, I've noticed a lot of ugliness that I want to fix.<br />
<br />
Everything here is assuming Python as the language of choice:<br />
<ul>
<li>For scripts that send email, check to make sure the list of mail receivers is up-to-date.</li>
<li>Look for those nasty embedded IP addresses and replace them with DNS names. </li>
<li>Change from old-style <a href="http://docs.python.org/2/library/stdtypes.html#file-objects" target="_blank"><i>open(FILE)/close(FILE)</i> constructs to <i>with open(FILE) as f</i> constructs</a>.</li>
<li>Get rid of "pickles" for persistent structured data storage. <a href="http://docs.python.org/2/library/pickle.html" target="_blank">Pickles</a> are a form of native Python object serialization that are quick and convenient, but have a lot of potential problems. I've mostly used Python's native <a href="http://docs.python.org/2/library/sqlite3.html" target="_blank">SQLite3</a> library to store data in simple persistent databases, but occasionally I just use plain text files.</li>
<li>Look for repetitive code in the main script logic and try to move it into functions or classes where possible. For example, I had several scripts that were building email reports via clunky string concatenation, so I created a Report class that knows how to append to itself and do simple formatting.</li>
<li>Remove unused module imports (that were usually there for debugging).</li>
<li>Standardize module imports, declaration of constants, etc.</li>
<li>Add more comments!</li>
<li>Remove old code that was commented out for whatever reason.</li>
<li>Look for places where I was creating huge lists in memory and try to figure out a way to reduce memory consumption with <a href="http://docs.python.org/2/reference/expressions.html?highlight=generator#generator-iterator-methods" target="_blank">generators</a>.</li>
</ul>
Things I haven't done, but probably should:<br />
<ul>
<li>Change to new-style <a href="http://docs.python.org/2/library/string.html#format-specification-mini-language" target="_blank">string formatting</a>.</li>
<li>Write some tests, particularly for log parsers.</li>
<li>Migrate from <a href="http://docs.python.org/2/library/optparse.html" target="_blank">optparse</a> to <a href="http://docs.python.org/2/library/argparse.html" target="_blank">argparse</a> for handling CLI flags. </li>
</ul>
Things I haven't done, and probably never will:<br />
<ul>
<li> Migrate to <a href="http://alexgaynor.net/2013/dec/30/about-python-3/" target="_blank">Python 3</a>.</li>
</ul>
Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com1tag:blogger.com,1999:blog-5029689981158113588.post-84601862096961542992013-11-08T16:07:00.000-07:002013-11-08T16:07:16.141-07:00Handy Tshark ExpressionsTshark is the CLI version of Wireshark, and it's amazing. I'm going to start collecting some of my favorite tshark one-liners here. Check back often.<br />
<br />
<b>Find All Unique Filenames Referenced in SMB2 </b><br />
<span style="font-family: "Courier New",Courier,monospace;">tshark -r file.pcap -Tfields -e ip.src -e ip.dst -e text smb2 | grep -oP "GUID handle File: .*?," | sort | uniq | awk -F: '{print $2}' | sed 's/,//'</span><br />
<br />
Notes:<br />
You don't actually need to include the ip.src and ip.dst fields, since they're not extracted by the grep command. I include them in case I want to do an ad-hoc grep for an IP address during the analysis process. Another way to do the same thing would be to modify the display filter to look only for certain addresses, e.g.:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">tshark
-r file.pcap -Tfields -e text smb2 and ip.addr==1.1.1.1 | grep -oP "GUID
handle File: .*?," | sort | uniq | awk -F: '{print $2}' | sed 's/,//'</span>Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com1tag:blogger.com,1999:blog-5029689981158113588.post-36505207146880135532013-11-01T19:41:00.000-06:002013-11-01T19:41:53.769-06:00How to Tell if TCP Payloads Are Identical<span style="font-family: inherit;">I was working on a problem today in which vendor tech support was suggesting that a firewall was subtly modifying TCP data payloads. I couldn't find any suggestion of this in the firewall logs, but seeing as how I've seen that vendor's firewall logs lie egregiously in the past, I wanted to verify it independently.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">I took a packet capture from both hosts involved in the conversation and started thinking about how to see if the data sent by the server was the same as the data received by the client. I couldn't just compare the capture files themselves, because elements like timestamps, TTLs, and IP checksums would be different.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">After a bunch of fiddling around, I came up with the idea of using tshark to extract the TCP payloads for each stream in the capture file and hash the results. If the hashes matched, the TCP payloads were being transferred unmodified. Here are the shell commands to do this:</span><br />
<br />
<span style="font-family: Courier New, Courier, monospace;">tshark -r server.pcap -T fields -e tcp.stream | sort -u | sed 's/\r//' | xargs -i tshark -r server.pcap -q -z follow,tcp,raw,{} | md5sum</span><br />
<span style="font-family: Courier New, Courier, monospace;">2cfe2dbb5f6220f29ff8aff82f7f68f5 *-</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: inherit;">You then run exactly the same commands on the "client.pcap" file and compare the resulting hashes. Let's break this down a bit more:</span><br />
<span style="font-family: inherit; font-size: x-small;"><br /></span>
<span style="font-family: 'Courier New', Courier, monospace;">tshark -r server.pcap -T fields -e tcp.stream</span><br />
<br />
This invokes tshark to read the "server.pcap" file and output the TCP stream indexes of each packet. This is just a long series of integers:<br />
<span style="font-size: x-small;"><br /></span>
0<br />
0<br />
1<br />
2<br />
1<br />
etc.<br />
<span style="font-size: x-small;"><br /></span>
The next command, <span style="font-family: Courier New, Courier, monospace;">sort -u</span>, produces a logical set of the unique (hence the "-u") stream indexes. In other words, it removes duplicates from the previous list. Not all Unix-like operating systems have the "<span style="font-family: Courier New, Courier, monospace;">sort -u</span>" option; if yours is missing it, you can use "<span style="font-family: Courier New, Courier, monospace;">| sort | uniq</span>" instead.<br />
<br />
Next,<span style="font-family: Courier New, Courier, monospace;"> sed 's/\r//'</span> removes the line break from the end of the resulting stream indexes. If you don't do this, you'll get an error from the next command.<br />
<span style="font-size: x-small;"><br /></span>
The next one's a bit of a doozy: <span style="font-family: Courier New, Courier, monospace;">xargs -i</span> takes each stream index (remember, these are just integers) and executes the <span style="font-family: 'Courier New', Courier, monospace;">tshark -r server.pcap -q -z follow,tcp,raw,{}</span>command once for each stream index, substituting the input stream index for the {} characters.<br />
<br />
The <span style="font-family: 'Courier New', Courier, monospace;">tshark -r server.pcap -q -z follow,tcp,raw,{} </span><span style="font-family: inherit;">command itself reads the capture file a second time, running the familiar "Follow TCP Stream in Raw Format" command from Wireshark on the specified TCP stream index that replaces the {} characters. If you're rusty on Wireshark, "Follow TCP Stream" just dumps the TCP payload data in one of a variety of formats, such as "raw" or ASCII. If you've never used this option in Wireshark, make sure you try it today!</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">The final command, </span><span style="font-family: Courier New, Courier, monospace;">md5sum</span><span style="font-family: inherit;">, runs a MD5 hash on the preceding input.</span><br />
<span style="font-family: inherit;"><br /></span>
To summarize, we've done this: taken a file, extracted all the raw TCP data payloads from its packets (without headers), and hashed the data with MD5. If we do this on two files and the hashes are the same, we know they contain exactly the same TCP data (barring the infinitesimally small probability of a MD5 hash collision).<br />
<br />
In my case, both capture files produced the same hash, proving that the firewall was (for once) playing nice.Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com2tag:blogger.com,1999:blog-5029689981158113588.post-87698896882974876132013-10-16T10:53:00.002-06:002013-10-16T11:02:39.654-06:00Java is to JavaScript as Car is to Carpet - a Beginner's GuideSome recent discussions at work have led me to the surprising realization that lots of people working in IT don't understand that Java and JavaScript are almost completely unrelated to each other. This is actually a fairly important misunderstanding to correct: it leads to wasted troubleshooting efforts, such as downgrading or upgrading Windows Java installations in response to browser JavaScript errors.<br />
<br />
I found the title of this blog entry in a <a href="http://stackoverflow.com/questions/245062/whats-the-difference-between-javascript-and-java" target="_blank">StackOverflow post</a>: "Java is to JavaScript as Car is to Carpet". That's pretty much it, in a nutshell. For the record, the only things that Java and JavaScript have in common are:<br />
<ol>
<li>They are both programming languages. </li>
<li>The word "Java".</li>
<li>Both came out of the web technology explosion of the early 1990s.</li>
<li>Both are frequently encountered in the context of web browsers.</li>
</ol>
<a href="https://en.wikipedia.org/wiki/Java_programming_language">Java </a>is a compiled programming language that was originally developed with a major goal of allowing similar or identical codebases to run on different platforms without needing to be recompiled. It does this by compiling to "bytecode" rather than platform-specific machine code, which then typically runs inside a so-called "Java Virtual Machine". Java was originally developed and controlled by Sun Microsystems (now Oracle), but it has since been re-licensed under the GNU Public License. Numerous open-source Java implementations now exist, but the Oracle/Sun version is still the most familiar to the average user.<br />
<br />
Java is associated with the web browser experience because of the widespread use of Java "applets" that are embedded in browser windows. Applets are not technically part of the browser; the compiled Java bytecode is downloaded by the browser and executed in a Java Virtual Machine (JVM) as a separate process. Applets are frequently transferred as a compressed "Java archive", or JAR file. Applets downloaded by a browser do not necessarily need to run in a browser window, but the fact that they are frequently embedded there leads to some confusion.<br />
<br />
Neither is Java necessarily a client-side technology: many popular server-side applications are written in Java and execute in a server-side JVM. Google's Android platform extends things even further, using Java as the programming language but compiling the bytecode to execute on their own proprietary virtual machine.<br />
<br />
<a href="https://en.wikipedia.org/wiki/Javascript">JavaScript</a>, on the other hand, is an interpreted (i.e., non-compiled) programming language that was originally developed to run inside web browsers. It was developed at Netscape and was later adopted by Microsoft and standardized as "ECMAScript". The use of "Java" in the name "JavaScript" was probably an attempt to piggyback on the popularity of Java; the two languages have almost nothing in common from a technical perspective.<br />
<br />
JavaScript is most frequently used to control the web browser experience, but there are many projects that use JavaScript completely outside the browser. My first experience with this dates back to the late 1990s, when I used a JavaScript-based commercial tool to automate software deployments to Windows workstations. Today, there are many interesting non-browser-embedded JavaScript platforms, such as <a href="http://nodejs.org/">Node.js</a> and <a href="https://github.com/ariya/phantomjs">PhantomJS</a>.Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com13tag:blogger.com,1999:blog-5029689981158113588.post-72403035189017395312013-08-02T16:43:00.000-06:002013-09-04T07:00:02.475-06:00Understanding Flow Export TerminologyThe variety of terms used to describe network flow export technologies and components can be pretty confusing. Just last year I wrote a post on <a href="http://unroutable.blogspot.com/2012/04/why-netflow-isnt-web-usage-tracker.html" target="_blank">web usage tracking and NetFlow</a> that is already a bit obsolete, so here's an attempt to explain some of the newer terms and capabilities in use today. <br />
<br />
<b>NetFlow Version 5</b><br />
NetFlow v5 is sort of the least common denominator in flow technologies. Almost all vendors and devices that support a flow export technology will do NetFlow v5. Because it's only capable of exporting information about packet fields up to layer 4, however, it's not flexible enough to use for analytics that require information about the application layer. NetFlow v5 tracks only the following data:<br />
<ul>
<li>Source interface</li>
<li>Source and destination IP address</li>
<li>Layer 4 protocol</li>
<li>TCP flags</li>
<li>Type of Service</li>
<li>Egress interface</li>
<li>Packet count</li>
<li>Byte count</li>
<li>BGP origin AS</li>
<li>BGP peer AS</li>
<li>IP next hop</li>
<li>Source netmask</li>
<li>Destination netmask</li>
</ul>
<b>Netflow Version 9</b><br />
Netflow v9 was Cisco's first attempt at defining an extensible flow export format, defined in <a href="https://www.ietf.org/rfc/rfc3954.txt" target="_blank">RFC 3954</a> back in 2004. It provides a flexible format for building customizable flow export records that contain a wide variety of information types. Many of the goals for flexible flow export were defined in <a href="https://tools.ietf.org/html/rfc3917" target="_blank">RFC 3917</a>:<br />
<ul>
<li> Usage-based accounting</li>
<li>Traffic profiling</li>
<li>Traffic engineering</li>
<li>Attack/Intrusion Detection</li>
<li>QoS monitoring</li>
</ul>
The RFC defines 79 field types that may be exported in NetFlow v9 packets, and directs the reader to the Cisco website for further field types. The <a href="http://www.cisco.com/en/US/technologies/tk648/tk362/technologies_white_paper09186a00800a3db9.html" target="_blank">latest document I could find</a> there defines 104 field types, several of which are reserved for vendor proprietary use and some of which are reserved for Cisco use.<br />
<br />
<b>IPFIX</b><br />
IPFIX is the IETF standard for extensible flow export. The basic protocol is specified in <a href="https://www.ietf.org/rfc/rfc5101.txt" target="_blank">RFC 5101</a>, but details are included in many other RFCs (Wikipedia has a <a href="https://en.wikipedia.org/wiki/IPFIX#External_links" target="_blank">partial list</a>). IPFIX is based directly on NetFlow v9 and is generally interoperable, but since it's an open standard it is extensible without Cisco involvement. Hundreds of field types are defined in the <a href="https://www.iana.org/assignments/ipfix/ipfix.xhtml" target="_blank">IANA IPFIX documentation</a>.<br />
<br />
<a href="https://datatracker.ietf.org/doc/rfc6759/?include_text=1" target="_blank">RFC 6759</a> defines an extension of IPFIX to include application-specific information in IPFIX export packets. This allows deep-packet-inspection technologies (such as Cisco's NBAR) to send information about non-standardized, tunneled, or encrypted application layer protocols to IPFIX collectors.<br />
<br />
IPFIX is being used by various vendors (Plixer, Lancope, and nProbe/nTop come to mind) to export HTTP header data, making it capable of being used as a web usage tracker or web forensics tool with the appropriate collector/analyzer software. <br />
<br />
<b>Flexible NetFlow</b><br />
As far as I can tell, Flexible NetFlow is a marketing term used by Cisco to encompass everything about their approach to configuring and implementing NetFlow v9 and IPFIX.<br />
<br />
<b>NSEL (NetFlow Security Event Logging)</b><br />
NSEL is a proprietary extension of NetFlow v9 to used by Cisco's ASA firewalls to export firewall log data. It's not clear to me why Cisco didn't use IPFIX for this purpose.<br />
<br />
<b>Cisco AVC (Application Visibility and Control)</b><br />
AVC is another Cisco marketing term that encompasses a variety of technologies surrounding the DPI and application-based routing capabilities in its routers, such as IPFIX, NetFlow v9, NBAR, PfR, ART (Application Response Time), and more.<br />
<br />
<b>Other Vendors</b><br />
As mentioned above, most network technology vendors support NetFlow v5 and/or v9. IPFIX support is now becoming very common. Some vendors use proprietary extensions of NetFlow v9; Riverbed's CascadeFlow is one example of this.<br />
<br />
In a followup post, I'll take a look at some tools that produce flow data without using export technologies.<br />
<br />Jay Swanhttp://www.blogger.com/profile/02571029118821999072noreply@blogger.com1