Tuesday, July 1, 2014

Simple Python Syslog Counter

Recently I did a Packet Pushers episode about log management. In it, I mentioned some of the custom Python scripts that I run to do basic syslog analysis, and someone asked about them in the comments.

The script I'm presenting here isn't one of the actual ones that I run in production, but it's close. The real one sends emails, does DNS lookups, keeps a "rare messages" database using sqlite3, and a few other things, but I wanted to keep this simple.

One of the problems I see with getting started with log analysis is that people tend to approach it like a typical vendor RFP project: list some requirements, survey the market, evaluate and buy a product to fit your requirements. Sounds good, right? The problem with log analysis is that often you don't know what your requirements really are until you start looking at data.

A simple message counting script like this lets you look at your data, and provides a simple platform on which you can start to iterate to find your specific needs. It also lets us look at some cool Python features.

I don't recommend pushing this too far: once you have a decent idea of what your data looks like and what you want to do with it, set up Logstash, Graylog2, or a similar commercial product like Splunk (if you can afford it).

That said, here's the Python:

I tried to make this as self-documenting as possible. You run it from the CLI with a syslog file as the argument, and you get this:

$ python simple_syslog_count.py sample.txt
 10    LINK-3-UPDOWN
 2     SSH-5-SSH2_CLOSE


     2     LINK-3-UPDOWN

[Stuff deleted for brevity]

For Pythonistas, the script makes use of a few cool language features:

Named, Compiled rRgexes

  • We can name a regex match with the (?PPATTERN) syntax, which makes it easy to understand it when it's referenced later with the .group('') method on the match object.
  • This is demonstrated in lines 36-39 and 58-59 of the gist shown above. 
  • It would be more efficient to capture these fields by splitting the line with the .split() string method, but I wanted the script to work for unknown field positions -- hence the regex. 

Multiplication of Strings

  • We control indentation by multiplying the ' ' string (that a single space enclosed in quotes) by an integer value in the print_counter function (line 50).
    • The reason this works is that the Python str class defines a special __mul__ method that controls how the * operator works for objects of that class:
      >>> 'foo'.__mul__(3)
      >>> 'foo' * 3

collections.Counter Objects

  • Counter objects are a subclass of dictionaries that know how to count things. Jeremy Schulman talked about these in a comment on the previous post. Here, we use Counters to build both the overall message counts and the per-device message counts:
>>> my_msg = 'timestamp ip_address stuff %MY-4-MESSAGE:other stuff'
>>> CISCO_MSG = re.compile('%(?P.*?):')
>>> from collections import Counter
>>> test_counter = Counter()
>>> this_msg = re.search(CISCO_MSG,my_msg).group('msg')
>>> this_msg
>>> test_counter[this_msg] += 1
>>> test_counter
Counter({'MY-4-MESSAGE': 1})

collections.defaultdict Dictionaries

  • It could get annoying when you're assigning dictionary values inside a loop, because you get errors when the key doesn't exist yet. This is a contrived example, but it illustrates the point:

    >>> reporters = {}
    >>> for reporter in ['','']:
    ...     reporters[reporter].append['foo']
    Traceback (most recent call last):
      File "", line 2, in
    KeyError: ''

  • To fix this, you can catch the exception:

    >>> reporters = {}
    >>> for reporter in ['','']:
    ...     try:
    ...         reporters[reporter].append['foo']
    ...         reporters[reporter].append['bar']
    ...     except KeyError:
    ...         reporters[reporter] = ['foo']
    ...         reporters[reporter].append('bar')
  • As usual, though, Python has a more elegant way in the collections module: defaultdict
>>> from collections import defaultdict
>>> reporters = defaultdict(list)
>>> for reporter in ['','']:
...     reporters[reporter].append('foo')
...     reporters[reporter].append('bar')
>>> reporters
defaultdict(, {'': ['foo', 'bar'], '': ['foo', 'bar']})
In the syslog counter script, we use a collections.Counter object as the type for our defaultdict. This allows us to build a per-syslog-reporter dictionary that shows how many times each message appears for each reporter, while only looping through the input once (line 66):

 per_reporter_counts[reporter][msg] += 1

Here, the dictionary per_reporter_counts has the IPv4 addresses of the syslog reporters as keys, with a Counter object as the value holding the counts for each message type:

>>> from collections import Counter,defaultdict
>>> per_reporter_counts = defaultdict(Counter)
>>> per_reporter_counts['']['SOME-5-MESSAGE'] += 1
>>> per_reporter_counts
defaultdict(, {'': Counter({'SOME-5-MESSAGE': 1})})
>>> per_reporter_counts['']['SOME-5-MESSAGE'] += 5
>>> per_reporter_counts
defaultdict(, {'': Counter({'SOME-5-MESSAGE': 6})})

If you got this far, you can go implement it for IPv6 addresses. :-)


Paul Stewart said...

It is very cool to be able to do that in Python. I certainly need to pick up some skills there and this serves as a good example. I currently do something very similar to this by chaining together awk, count and sort. Thanks for sharing.

cloudtechnerd said...

I know a lot of people swear by Python but is it really better than its' competitors?


Rob said...

Nice syslog counter, thanks! Python is great for math stuff in my experience.

Gokul Ravi said...

very nice interview questions
vlsi interview questions
extjs interview questions
laravel interview questions
sap bi/bw interview questions
pcb interview questions
unix shell scripting interview questions

Gokul Ravi said...

really awesome blog
hr interview questions
hibernate interview questions
selenium interview questions
c interview questions
c++ interview questions
linux interview questions

Gokul Ravi said...

thanks for sharing this blog
spring mvc interview questions
machine learning online training
servlet interview questions mytectra.in
wcf interview questions

Gokul Ravi said...

nice blog
android training in bangalore
ios training in bangalore

Suresh R said...

Iot Training in Bangalore
Artificial Intelligence Training in Bangalore
Machine Learning Training in Bangalore
Blockchain Training bangalore
Data Science Training in Bangalore
Big Data and Hadoop Training in bangalore
Devops Training in Bangalore