Blog

You may disagree with a lot of things President Reagan said, but you can’t deny the utility of one of his favorite phrases, “Trust, but verify”.  Log files, like spy planes, keep honest people honest. Logs can provide proof of unauthorized access, catching employees or others doing things they shouldn’t be doing on your network. In addition, your logs are often your only recourse to figure out what went wrong and when on a system that is heading south.

The problem is, any smart trespasser knows to cover his tracks.  The last thing he or she will do is to “clean up” the system logs to erase everything they did.  Entries can be selectively parsed out to keep things looking normal.  If someone gains root access to your server, there’s little you can do to stop them hiding their tracks after planting malicious scripts or binaries for you to unwittingly execute after they are long gone.

Using a central logging server, you can capture bad behavior until the intruder figures out he’s being logged elsewhere.  If the logging server is set up properly, there’s little they can do to erase what they’ve already done, and it might just be enough to incriminate them.  In addition, if you are maintaining hundreds or perhaps thousands of systems out there, a central log server can provide a comprehensive account of what’s happening on your network.  With everything in one place, it’s easy to digest the massive amount of data using commercial products such as Splunk or even grep to watch trends and weed out the unimportant stuff.

We’ll configure a very basic central syslog server using the standard udp syslog utility.  There are more sophisticated solutions out there, such as syslog-ng,  which we will cover later. More advanced systems can provide encrypted communications to the log server, along with a host of other improvements.

Configuring the syslog server

Since your goal is to have your logs secured and untampered, it obviously pays to have a purpose-built, locked-down syslog server.  There shouldn’t be much else running on the box, and only a select few people should have access to it.

Almost every unix/linux system in existence has a syslog daemon process running on it.  However, you have to make some config changes for it to accept outside syslog communication. On Linux, this is accomplished using the -r switch at startup.  In Red Hat Enterprise Linux, this option is set in /etc/sysconfig/syslog:

SYSLOGD_OPTIONS="-r -m 0"

You’ll need to restart the syslog daemon after making this change:

/sbin/service syslogd restart

And then double-check it’s listening for incoming messages:

bash-3.2# netstat -ntaupe | grep syslog

udp    0   0 0.0.0.0:514    0.0.0.0:*    0    1019505359 12837/syslogd

Looks good. It’s a good idea to limit connections to your internal, trusted systems.  Otherwise in its default config, syslog will happily accept anything coming in on udp 514 from any location.

Using Netfilter, we can limit connections to a commonly-used RFC 1918 address space of 254 machines (assuming a default of DROP/DENY):

iptables -A my-input-chain -m state --state NEW -m udp -p udp -s 192.168.1.0/24 --dport 514 -j ACCEPT

Configuring the syslog client

Configuration is quite easy on the client side.  We just need to edit /etc/syslog.conf:

# Send all logs to the syslog server
*.*	@logs.pretendco.com

And then restart syslogd again:

/sbin/service syslog restart

This will allow logs to continue to be written locally, as well as sending logs to the central server.

To test that everything’s working, there is a handy utility called logger, which submits messages to the syslog facility.  You can specify the facility and level of the message to check your syslog filters are working as expected:

/bin/logger -p facility.level -t 'test from logger' 'Jello, World'

That’s all there is to it! The next step is to carefully define your facility and level entries in

/etc/syslog.conf

. This will keep your central log server’s default messages file from being overrun with a bunch of junk.