HowTo: syslog-ng/stunnel central loghost

Print Friendly, PDF & Email

This howto is mainly for the purpose of documenting the approach I have taken to collecting logs produced on servers on a central syslog-ng loghost where they can later be analysed. Some of the info in this howto has been blatantly ripped off a couple of websites:

The first thing you will need is a host to act as the central loghost which boasts a reasonable amount of space for storage of your logs. I would also suggest that this machine is locked down as much as possible and restrict access to a minimum.

I will assume that you have installed the packages syslog-ng and stunnel on both the client server and the loghost.

And for the sake of this howto I’ll assume that the syslog server’s hostname is loghost.ilrt.bris.ac.uk. You’ll also need to make sure the firewall on the loghost is configured to allow connections to port 5140. Because we are using client certificate authentication in stunnel this will be safe to open to your network as a whole.

Syslog-ng configuration on the loghost

The important parts of the configuration for syslog-ng (held in /etc/syslog-ng/syslog-ng.conf) is as follows:

Setup a listening port on localhost for syslog-ng to connect to:
source s_stunnel {
tcp(ip(127.0.0.1) port(5000) max-connections(300));
};

Create a dynamic destination to create files/directories so that you do not need to rotate the log files.
destination df_host {
file("/var/log/host/messages/$R_YEAR/$R_MONTH/$R_YEAR-$R_MONTH-$R_DAY"
template("$ISODATE <$FACILITY.$PRIORITY> $HOST $MSG\n")
template_escape(no)
);
};

Add a filter to each local log directive (if required) to make sure the standard log files do not contain info about the network based machines.
filter f_localhost { host(localhost.*) or host(loghost.ilrt.bris.ac.uk); };

Finally add together the localhost’s sources with the network based sources and send that to the dynamic destination added above.
log {
source(s_local);
source(s_stunnel);
destination(df_host);
};

Full configuration for a Debian based machine.

Stunnel configuration and certificates

Next up you need to configure a stunnel daemon to listen on the external interface and send messages to the localhost syslog-ng server. So to be clear we’re going to have stunnel listening accepting encrypted connections on loghost.ilrt.bris.ac.uk:5140 passing unencrypted data to localhost:5000.

First of all you need to generate some certificates. We need a server side certificate and a client side certificate. We will ask stunnel to only make a connection if both are verified as correct from both server and client side. The logclient hostname set below is not a real hostname and it can be anything you like. I’m just using it for consistency.

mkdir -p /etc/ssl/syslog-ng
cd /etc/ssl/syslog-ng
for host in loghost logclient ;do
openssl req -new -x509 -nodes -days 999 -newkey rsa:2048 \
-subj "/C=GB/ST=Bristol/L=Bristol/O=University of Bristol/OU=ILRT/CN=$host.ilrt.bris.ac.uk" \
-out $host.pem -keyout $host.pem
dd if=/dev/urandom of=tmpstunnel count=2
openssl dhparam -rand tmpstunnel 512 >> $host.pem
ln -sf $host.pem `openssl x509 -noout -hash < $host.pem`.0
rm tmpstunnel
done

Then we need to startup the stunnel in daemon mode (preferably with an init script and a basic config file passing arguments to the script):

/usr/bin/stunnel -d 5140 -r 5000 -s stunnel -v 3 -p /etc/ssl/syslog-ng/loghost.pem -A /etc/ssl/syslog-ng/logclient.pem<code>

Strictly speaking the loghost doesn’t need to know the private key of the client so this can be removed from /etc/ssl/syslog-ng/logclient.pem on the loghost. Likewise the client doesn’t need to know the private key of the server so can be removed from /etc/ssl/syslog-ng/loghost.pem on the clients. If you’re security conscious then I’d recommend making 2 separate copies of the files for client and server with and without private keys where applicable.

Copy the files over to the client machines and run a similar command to make a connection to the loghost stunnel:

/usr/bin/stunnel -c -d 127.0.0.1:5000 -r loghost.ilrt.bris.ac.uk:5140 -s stunnel -v 3 -p /etc/ssl/syslog-ng/logclient.pem -A /etc/ssl/syslog-ng/loghost.pem

The differences here are that we are running stunnel in client mode to create a plain text source port to an encrypted remote port. We are also asking for the certificates to be verified but notice we have switched the files used between the arguments. Therefore we are supplying the local cert to be client and the remote cert to be verified against the server’s.

You should now be able to simply telnet to localhost on port 5000 type anything and it should appear in the relevant /var/log/host/messages/YEAR/DAY/DATE file. /var/log/messages on the local machine will indicate any certificate verification issues. If you get disconnected immediately then it’s most likely the remote stunnel isn’t running or has it’s listen/remote ports set incorrectly. If you start the telnet and it just sits there it’s probably a firewall issue.

Syslog-ng config on the client servers (that’s everything)

Next we need a local syslog-ng.conf. The important parts of this configuration are:

Setup some good defaults for sending messages over the network.options {
chain_hostnames(no);
time_reopen(10);
sync(0);
...
};

Loghost via local stunnel connection
destination dh_stunnel { tcp("localhost" port(5000)); };

Send everything to the stunnel destination
log {
source(s_local);
destination(dh_stunnel);
};

syslog-ng.conf for logging clients

After you start the syslog-ng daemon you should be able to send log messages to the central loghost:

logger “Hello World!”