Automated Log Monitoring with LogSentry and a Central Syslog Server

Part One: Installing and Configuring LogSentry

by Urbana Der Ga’had

If you haven’t automated the monitoring of your system logs, you’re flying blind. By proactively identifying hardware failures, unusual patterns of user behavior, and intrusion attempts, you can often solve problems before they become painful. festering. scars that won’t heal. Oh yes! In Part One of this article we will install and configure LogSentry(formerly logcheck) on a stand-alone Red Hat Linux system. In Part Two we will set up a central syslog server, centralize the log checking, and configure the client systems to use the central syslog server. In Part Three we will look at ways to further enhance the security and performance of this setup.

First go visit the fine folks at Psionic, who also produce PortSentry and HostSentry, two other great security products. LogSentry is free and now licensed under the GPL.

Download LogSentry

Untar and build the software.

bash-2.04# tar xvzf logsentry-1.1.1.tar.gz
bash-2.04# make linux
make install SYSTYPE=linux
make[1]: Entering directory `/usr/local/src/logcheck-1.1.1'
Making linux
cc -O -o ./src/logtail ./src/logtail.c
./src/logtail.c: In function `main':
./src/logtail.c:51: warning: return type of `main' is not `int'
Creating temp directory /usr/local/etc/tmp
Setting temp directory permissions
chmod 700 /usr/local/etc/tmp
Copying files
cp ./systems/linux/logcheck.hacking /usr/local/etc
cp ./systems/linux/logcheck.violations /usr/local/etc
cp ./systems/linux/logcheck.violations.ignore /usr/local/etc
cp ./systems/linux/logcheck.ignore /usr/local/etc
cp ./systems/linux/logcheck.sh /usr/local/etc
cp ./src/logtail /usr/local/bin
Setting permissions
chmod 700 /usr/local/etc/logcheck.sh
chmod 700 /usr/local/bin/logtail
chmod 600 /usr/local/etc/logcheck.violations.ignore
chmod 600 /usr/local/etc/logcheck.violations
chmod 600 /usr/local/etc/logcheck.hacking
chmod 600 /usr/local/etc/logcheck.ignore
Done. Don't forget to set your crontab.
make[1]: Leaving directory `/usr/local/src/logcheck-1.1.1'

By default the software will be installed in /usr/local/etc. Edit logcheck.sh to configure the user who should receive the logcheck reports. Search for:

SYSADMIN=root

Change this variable to some appropriate alias that you use to receive security related mail. You can also check out the command paths in this script to make sure they’re ok for your system, they have been fine on our Red Hat system.

Add a cron job, using either the crontab command to edit root’s crontab (in /var/spool/cron/) or adding an entry in /etc/crontab if you prefer. I am old-fashioned so I use crontab to edit the old style crontab.
Add a line like this:

0,30 * * * * /usr/local/etc/logcheck.sh

We recommend running the logcheck script at least every thirty minutes.

Now you can start customizing the filters. Do peruse the README files that come with LogSentry. There are excellent tips about how to configure syslog, what the keywords and filters mean and suggestions as to how one should respond to security violations.

[root@goblin etc]# ls -al
total 44
drwxr-xr-x    4 root     root         4096 Jun  4 08:45 .
drwxr-xr-x   22 root     root         4096 May 26 16:36 ..
-rw-------    1 root     root         1037 Jun  4 08:45 logcheck.hacking
-rw-------    1 root     root         1172 Jun  4 08:45 logcheck.ignore
-rwx------    1 root     root        10633 Jun  4 08:45 logcheck.sh
-rw-------    1 root     root          407 Jun  4 08:45 logcheck.violations
-rw-------    1 root     root           14 Jun  4 08:45 logcheck.violations.ignore

The files logcheck.hacking and logcheck.violations contain patterns which trigger security alerts when matched. The files logcheck.violations.ignore and logcheck.ignore contain patterns which will be ignored when found in the logs.
There are three levels of alert in the logcheck reports. “ACTIVE SYSTEM ATTACK” will include messages which matched patterns in logcheck.hacking. “Security Violations” will include anything which matched patterns in logcheck.violations and did not match when reverse checked against logcheck.violations.ignore. “Unusual System Activity” will include anything found in the logs which did not match an entry in logcheck.ignore, the catch-all file for patterns which are safe to ignore. This is the file you want to work with, following the theorem of “that which is not explicitly permitted is forbidden.”
Once you’ve got LogSentry installed and the cron job set up to run it periodically, you will start getting a bunch of mail. This is how you start to determine which messages are safe to ignore. Be careful to make the strings as specific as possible, you want to control very closely what is ignored. As an example, we are going to exclude some ordinary pop3 messages from the alerts. These log entries came from /var/log/maillog, and were sent to us in our logsentry report. We will paste the entry from the report into logcheck.ignore, and edit the line to make it a valid pattern for logsentry to match.

for example:

Jun  2 04:06:17 goblin ipop3d[256]: Login user=cindy host=hop.pop.com [1.2.3.3] nmsgs=0/0
Jun  2 04:06:18 goblin ipop3d[256]: Logout user=cindy host=hop.pop.com [1.2.3.3] nmsgs=0 ndele=0

becomes:

goblin ipop3d.*: Login user=cindy host=hop.pop.com [1.2.3.3] nmsgs=.*
goblin ipop3d.*: Logout user=cindy host=hop.pop.com [1.2.3.3] nmsgs=.*

Notice the use of the wildcard .* to match 0 or more characters, also special characters such as () or [] should be escaped with a backslash. Also, be sure there are *no* blank lines in your logcheck.ignore or logcheck.violations.ignore files, because that will cause logsentry to ignore everything! Not what you want.

Next Time: let’s centralize control.