Splunk> Making Sense of Logfiles

My area of expertise in the professional world is Network Security. This includes protecting network from intrusions, but also delivering reports about the network status. For the latter we use SIEM(like) environments like the Cisco CS-MARS and the Juniper STRM.
The 'problem' with these devices is that they are great in reporting incidents and creating awesome reports about everything, but they lack the functionality to do some serious investigating.

I have several customers with a SIEM, and most of them still use (Linux) command-line tools like awk, grep, etc. these tools work, but you need to scrape everything together yourself, and building queries can be quite challenging. This is where Splunk> comes in.

Splunk> is a AJAX/Flash enabled search-engine (much like Google), that gives you an easy way of digging through numerous log-files. And the great part is, it's free (with some limitations). And because it's free, it's great for home usage. Especially if you're running a firewall and some services at home (like a webserver, SSH, Database, etc). Logging for all these services are combined on the Splunk> server (it doesn't have to be a dedicated server), and you get a user friendly interface for digging through the logs.Splunk> Search InterfaceThe great thing is that you can dig deeper-and-deeper through the logs with every mouse click. It's real easy to see user activity in a certain time-frame (or even real-time). The following screenshot shows the user account 'willem' in combination with the SSH daemon (sshd).

Search results for 'willem' and 'sshd' in the last 24 hoursFrom this point on you can refine your search, dig deeper, etc. When you know how to use Google, you can also use Splunk>.

Installing and Running Splunk>

Before you can use it, you need to install it. The installation is straight forward on Windows and Linux. Not many surprises on that end. The OS X installation is a bit off. At least in my case. So the next couple of paragraphs are dedicated to (my) installation problems on a Mac mini with Apple OS X Server 10.6.4.

After installing Spunk from the downloaded DMG file you need to start it. This is done from the Terminal.

cd /Applications/splunk/bin/
./splunk start

If it's the first time you start Splunk>, you need to agree to the license mumbo-jumbo. After that the service should start, and you'll see output similar to the one displayed below;

zeus:bin administrator$ ./splunk start

Splunk> Take the sh out of IT.

Checking prerequisites...
    Checking http port [8000]: open
    Checking mgmt port [8089]: open
    Checking configuration...  Done.
    Checking index directory...  Done.
    Checking databases...
    Validated databases: _audit, _blocksignature, _internal, _thefishbucket, history, main, sample, splunklogger, summary
All preliminary checks passed.

Starting splunk server daemon (splunkd)... Done.
Starting splunkweb... Done.

If you get stuck, we're here to help.  
Look for answers here: http://www.splunk.com/base/Documentation

The Splunk web interface is at http://zeus.local:8000

Now you're ready to Splunk>. If that didn't happen, you may continue to read the troubleshootin part a bit further on.

To start Splunk> at boot time, you need to add the startup script to the 'global' startup items. To make things easy, just enter the following;

zeus:bin administrator$ sudo ./splunk enable boot-start
Password:
Init script installed at /System/Library/StartupItems/Splunk/Splunk.
Init script is configured to run at boot.
zeus:bin administrator$

Accessing and Configuring Splunk>

Splunk> runs on its own webserver, so no need to enable Apache or what ever. Basic configuration (like adding the logsources) is done through the webinterface. Note that upon initial installation, the service runs in an evaluation mode. This means that you get full functionality for the first 30 days. After that, you need to get a license, or the service become the limited free version.

Access the server by entering the following in your browser:

http://<hostname>:8000

The trail version had username and password authentication (u:admin / p:changeme). One of the main differences is that the free version doensn't have any user account, so be carefull with connecting your Splunk> service to the internet. The first couple of days, it's relatively secure, but after that, anyone can access it. So leave it accessible only to the local LAN (or use a reverse proxy with authentication).

To add additional datasources (e.g. local log files) go to the Manager part in Splunk> and select Data inputs. Here you can add local files and folder, but also listeners.

Splunk> Data inputsListeners can receive e.g. syslog and snmp traffic. This way, your filrewall or other devices can also send their logs to Splunk>. If you are running a Windows server you can use an Eventviewer to Syslog interface. I played around with the Snare Agent for Windows by Intersect Alliance, which works perfectly.

There are viritual no limitations on the types of logfiles. As long as it is plain text (ASCII), Splunk> can handle it. After adding the logfile locations and/or listeners, Splunk> will start to index the logfiles. How long this takes, depends on the amount of logging available, but on average you're good to go after a couple of minutes (go and have a coffee).

Troubleshooting

After installing Splunk> on several other operating systems, I thought that the OS X edition shouldn't be a problem. Well, Splunk> 4.1.3 (80534) has the habit of forgetting things during the installation process. It forgot (in my case) to add an environment variable to the OS. It seems that the following variable needs be be set;

$SPLUNK_HOME=/Applications/splunk/
export SPLUNK_HOME

I added this variable in my bash profile (~/.bash_profile). Restarting the Terminal allowed me to continue to start Splunk>.

If you're running the Splunk daemon at boot time, you may need to add these lines to /etc/profile or /etc/bashrc

And now on to the next challenge;

If the Terminal output is something like the following, you're in 'UTF8-trouble'.

zeus:bin administrator$ ./splunk start

Splunk> CSI: Logfiles.

Checking prerequisites...
    Checking http port [8000]: open
    Checking mgmt port [8089]: open
Traceback (most recent call last):
  File "/Applications/splunk/lib/python2.6/site-packages/splunk/clilib/cli.py", line 18, in <module>
    import control_api as ca
  File "/Applications/splunk/lib/python2.6/site-packages/splunk/clilib/control_api.py", line 26, in <module>
    import i18n
  File "/Applications/splunk/lib/python2.6/site-packages/splunk/clilib/i18n.py", line 7, in <module>
    from babel.messages import frontend
  File "/Applications/splunk/lib/python2.6/site-packages/babel/messages/__init__.py", line 16, in <module>
    from babel.messages.catalog import *
  File "/Applications/splunk/lib/python2.6/site-packages/babel/messages/catalog.py", line 30, in <module>
    from babel.dates import format_datetime
  File "/Applications/splunk/lib/python2.6/site-packages/babel/dates.py", line 28, in <module>
    from babel.numbers import get_decimal_symbol
  File "/Applications/splunk/lib/python2.6/site-packages/babel/numbers.py", line 41, in <module>
    LC_NUMERIC = default_locale('LC_NUMERIC')
  File "/Applications/splunk/lib/python2.6/site-packages/babel/core.py", line 642, in default_locale
    return '_'.join(filter(None, parse_locale(locale)))
  File "/Applications/splunk/lib/python2.6/site-packages/babel/core.py", line 763, in parse_locale
    raise ValueError('expected only letters, got %r' % lang)
ValueError: expected only letters, got 'utf-8'
zeus:bin administrator$

This means that there is UTF8 conflict on your system (don't ask....). This can be solved by adding a parameter to the splunk-launch.conf file. Just add the following to the file located in $SPLUNK_HOME/etc/

LC_CTYPE=

After this Splunk> should start normally.

The errors I ran into, seem to be for this specific version of Splunk> (which is a couple of days old at the time of writing this blogpost), since I didn't have any problems with an earlier version (or I must have blocked that completely).

Big thanks to SMT in supplying the troubleshooting answers.