Rsyslog on CentOS success story

From rsyslog wiki
Jump to: navigation, search

Contents

Pre-amble

This document explains how to install rsyslog on Red Hat derived systems by either using yum or downloading and installing an RPM manually. The instructions for using yum were added after this (extensive) document was written specifically for the manual RPM method only, so the flow might not be perfect, but you should be able to use EITHER the yum section or the few RPM related sections without much trouble.

The reason I started to look at rsyslog as an alternative to the stock syslog, was that I needed support staff to look at maillog entries when diagnosing and trouble-shooting email problems for clients. The best way to do this, without giving staff access to any shell account on the various mail hosts, was to get those mail hosts to log to a central MySQL server and allow access to that server from a web browser.

These instructions detail step-by-step how I performed this using (originally) rsyslog 1.17.5 running on Scientific Linux 4.5 (a Red Hat Enterprise Linux 4 Update 5 derivative) and using phpLogCon as the web interface to the MySQL database. I'm now using the latest rsyslog versions and will continue to update as new releases become made available.

These instructions should hold true not just for Scientific Linux, but for any RHEL-based derivative including CentOS, Tao, Whitebox Linux, etc as they all basically use the Red Hat SRPMs. These instructions also hold true for any rsyslog version up from 1.17.5 to 1.19.x.

Purpose of this document

To outline the installation instructions for rsyslog on Red Hat derived systems such as CentOS 5.x and Scientific Linux 4.5

Resources

The rsyslog homepage is here:

http://www.rsyslog.com

Original spec was obtained from:

http://people.redhat.com/pvrabec/rpms/rsyslog.spec

however pvrabec doesn't maintain this spec so I used it as a basis and updated accordingly on each rsyslog upgrade.

Using yum

Check if the yum repositories you have access to include any rsyslog packages:

yum search rsyslog
yum info rsyslog

The following is based on the rsyslog version 2.0.6 package that is in the CentOS base yum repository as of September, 2009. PLEASE NOTE that if version 3 or above becomes available, installation may be different due to the changes in rsyslog as of version 3.

Simply install rsyslog using yum:

yum install rsyslog

This should not do anything to your current (usually syslog/sysklogd) system. It just installs rsyslog in parallel (and it's up to you to turn it on). Everything is put in place for you (startup scripts, config file that is a mirror of syslog.conf, integration with log rotation, etc.) and you just have to:

chkconfig syslog off
chkconfig rsyslog on
service syslog stop
service rsyslog start

If you want to permanently remove syslog/sysklogd from your system, you can try:

chkconfig --del syslog
yum remove sysklogd

Note yum should also add rsyslog to /etc/logrotate.d/syslog so you don't have to touch any of this. After installation, that file should look similar to this:

/var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron {
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
        /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Leaving the syslog line in that file won't hurt anything.

You are now finished and have a working rsyslog replacement for syslog/sysklogd. You can now go to /etc/rsyslog.conf if you want to add more powerful logging rules.

Spec files

All spec files I maintain can be found here:

ftp://ftp.npgx.com.au/mmansour/specs

Currently I have the following spec files available:

ftp://ftp.npgx.com.au/mmansour/specs/rsyslog_v1.17.5.spec

ftp://ftp.npgx.com.au/mmansour/specs/rsyslog_v1.18.0.spec

ftp://ftp.npgx.com.au/mmansour/specs/rsyslog_v1.18.1.spec

ftp://ftp.npgx.com.au/mmansour/specs/rsyslog_v1.18.2.spec

ftp://ftp.npgx.com.au/mmansour/specs/rsyslog_v1.19.0.spec

ftp://ftp.npgx.com.au/mmansour/specs/rsyslog_v1.19.1.spec

ftp://ftp.npgx.com.au/mmansour/specs/rsyslog_v1.19.2.spec

ftp://ftp.npgx.com.au/mmansour/specs/rsyslog_v1.19.3.spec

ftp://ftp.npgx.com.au/mmansour/specs/rsyslog_v1.19.7.spec

ftp://ftp.npgx.com.au/mmansour/specs/rsyslog_v1.19.9.spec

ftp://ftp.npgx.com.au/mmansour/specs/rsyslog_v1.19.10.spec

For the purposes of saving me time in the future, I've submitted my spec file to the excellent rpmforge repository and hope someday they can take over and maintain it.

Note 1: although all these spec files are configured with MySQL enabled (the "--enable-mysql" setting in the spec file), they can still be used even if you're not using MySQL logging.

Note 2: if you're using rsyslog v1.19.0 and above, read the bottom of the wiki page for special instructions on the 1.19.x logging to MySQL.

Note 3: after reviewing my instructions from scratch, I discovered that I also needed the files:

ftp://ftp.npgx.com.au/mmansour/specs/rsyslog.init

ftp://ftp.npgx.com.au/mmansour/specs/rsyslog.sysconfig

in the /usr/src/redhat/SOURCES directory to properly build from the spec files provided.

Build RPM(s)

Once one of the spec files above have been downloaded and copied into:

        /usr/src/redhat/SPECS

and the accompanying rsyslog tarball (that the spec file references) has been downloaded and copied into:

        /usr/src/redhat/SOURCES

then it's simply a matter of doing:

        # rpmbuild -bb /usr/src/redhat/SPECS/<specfilename>

to build the rsyslog binary RPM's.

Install the newly built rsyslog RPM

        # rpm -Uvh /usr/src/redhat/RPMS/i386/rsyslog-1.17.5-1.i386.rpm
        Preparing...                ########################################### [100%]
           1:rsyslog                ########################################### [100%]
        warning: /etc/logrotate.d/syslog saved as /etc/logrotate.d/syslog.rpmsave

Note: The installation of this RPM will remove your old syslog setup, as we want it to do, and replace it with rsyslog.

Start rsyslog

By default rsyslog will start on boot:

        # chkconfig --list rsyslog
        rsyslog         0:off   1:off   2:on    3:on    4:on    5:on    6:off

Let's start rsyslog:

        # service rsyslog start
        Starting system logger (rsyslog):                          [  OK  ]
        Starting kernel logger (rklogd):                           [  OK  ]

and in the /var/log/messages file we see:

        Aug  7 20:09:24 servername1 rsyslogd: [origin software="rsyslogd" swVersion="1.17.5" x-pid="25699"]
        [x-configInfo udpReception="No" udpPort="514" tcpReception="No" tcpPort="0"] restart
        Aug  7 20:09:24 servername1 rsyslog: rsyslogd startup succeeded
        Aug  7 20:09:24 servername1 kernel: rklogd 1.17.5, log source = /proc/kmsg started.
        Aug  7 20:09:24 servername1 rsyslog: rklogd startup succeeded
        Aug  7 20:05:34 servername1 syslog: syslogd shutdown succeeded#012#000

Configure MySQL logging

Setup instructions are here:

http://www.rsyslog.com/module-Static_Docs-view-f-rsyslog_mysql.html.phtml

From that URL we see:

The schema definition is contained in the file "createDB.sql". It comes with the rsyslog package. Review it to check that the database name is acceptable for you.

So we need the tarball to extract this file from it (as I don't include this file in the RPM, I will modify the spec some day to include it):

        # tar zxvf rsyslog-1.18.0.tar.gz
        # cd rsyslog-1.18.0
        # ll *sql
        -rw-rw-r--  1 root root 1046 Jun 25 19:42 createDB.sql

Note: in rsyslog v1.19.10 this file can be found here ./plugins/ommysql/createDB.sql after tarball extraction.

We can use this file directly or, if you'd like to specify the database name (which I do):

        # vi createDB.sql

modify:

        CREATE DATABASE Syslog;
        USE Syslog;

to change the "Syslog" name to whatever database name you like.

Execute the createDB.sql file

Execute the createDB.sql on the MySQL database, which will produce the Syslog database and populate it with the relevant tables.

Assign MySQL account name and password to the Syslog database

For simplicity, I perform my MySQL work using phpMyAdmin, of course you can use whatever you choose including mysql command prompt. For security purposes, we’ll create a new MySQL username and password which will be used and displayed in the /etc/rsyslog.conf file.

Using phpMyAdmin, it is sufficient to grant this user INSERT privileges to the systemevents table only.

We also create the syslogwriter MySQL user and assign INSERT privileges to the systemevents table only.

The following syslogwriter MySQL accounts were created:

        syslogwriter@servername1.example.com.au
        syslogwriter@servername2.example.com.au
        syslogwriter@servername3.example.com.au
        syslogwriter@servernameN.example.com.au

up to N servers that will be logging into this MySQL database.

Note: the above MySQL accounts are needed for each server logging into a "central" MySQL server. If the MySQL server is not going to act as a central repository for syslog messages, then only syslogwriter@servername1.example.com.au is needed.

Tell rsyslog to log to the database

As explained in the URL above:

Next, we need to tell rsyslogd to write data to the database. As we use the default schema, we do NOT need to define a template for this. We can use the hardcoded one (rsyslogd handles the proper template linking). So all we need to do is add a simple selector line to /etc/rsyslog.conf

        *.*       >database-server,database-name,database-userid,database-password

So we make this change:

        # vi /etc/rsyslog.conf

add the following to the end of the file:

        *.*       >servername1.example.com.au,Syslog,syslogwriter,LongEncryPtedPaSsWord

Note: it’s important to use TAB spaces between entries above and to modify the values for your setup.

Tune DB performance

With big log databases, you will experience noticable performance drops in the web interface. To a degree, this can be compensated by adding a couple of indexes to the Syslog table, these are the ones used at my (Incase) site:

       KEY `FromHost` (`FromHost`),
       KEY `SysLogTag` (`SysLogTag`),
       KEY `Facility` (`Facility`),
       KEY `ReceivedAt` (`ReceivedAt`),
       KEY `DeviceReportedTime` (`DeviceReportedTime`),
       KEY `ProcessID` (`ProcessID`),
       FULLTEXT KEY `Message` (`Message`)

Secure the /etc/rsyslog.conf file

As discussed in the URL above:

As a side note, it is strongly advisable to make the rsyslog.conf file readable by root only - if you make it world-readable, everybody could obtain the password (and eventually other vital information from it).

So:

        # ll /etc/rsyslog.conf
        -rw-r--r--  1 root root 929 Aug  7 21:57 /etc/rsyslog.conf
        # chmod 640 /etc/rsyslog.conf

Restart rsyslog

        # service rsyslog restart
        Shutting down kernel logger (rklogd):                      [  OK  ]
        Shutting down system logger (rsyslog):                     [  OK  ]
        Starting system logger (rsyslog):                          [  OK  ]
        Starting kernel logger (rklogd):                           [  OK  ]

and:

        Aug  7 22:00:13 servername1 kernel: Kernel logging (proc) stopped.
        Aug  7 22:00:13 servername1 kernel: Kernel log daemon terminating.
        Aug  7 22:00:14 servername1 rsyslog: rklogd shutdown succeeded
        Aug  7 22:00:14 servername1 rsyslogd: [origin software="rsyslogd" swVersion="1.17.5" x-pid="25699"] 
        exiting on signal 15.
        Aug  7 22:00:14 servername1 rsyslogd: [origin software="rsyslogd" swVersion="1.17.5" x-pid="6774"]
        [x-configInfo udpReception="No" udpPort="514" tcpReception="No" tcpPort="0"] restart
        Aug  7 22:00:14 servername1 rsyslog: rsyslogd startup succeeded
        Aug  7 22:00:14 servername1 kernel: rklogd 1.17.5, log source = /proc/kmsg started.
        Aug  7 22:00:14 servername1 rsyslog: rklogd startup succeeded
        Aug  7 22:00:14 servername1 rsyslog: rsyslogd shutdown succeeded#012#000

At this point, we’re now logging into the MySQL database. To view the logs we need to use phpLogCon.

Install phpLogCon http://www.phplogcon.org/

phpLogCon is used to browse the MySQL database from any Web browser. It also supports creating user accounts for people to view those logged rsyslog entries. This is obviously extremely handy and the real power behind the viewing of the rsyslog entries on a central MySQL server.

The file can be downloaded from here:

http://www.phplogcon.org/Downloads-req-getit-lid-6.phtml

Extract the tarball

        # tar zxvf phplogcon-1.2.3.tar.gz

Create web path or virtual server

I normally create Apache virtual servers for any of these types of clustered setups, but you can just as easily copy the extracted phplogcon directory to:

        /var/www/html/phplogcon

if you choose.

Move phplogcon to web directory

        # mv phplogcon-v1.2.3/* /var/www/html/phplogcon

Run install.php for phplogcon

Prior to doing this, I had to assign ALL PRIVILEGES access to the Syslog database for the syslogwriter MySQL user otherwise the install.php script would not have the correct privileges to write to the tables. I didn't bother trying to work out exactly the privileges needed.

As explained on the URL:

http://new.phplogcon.org/index.php?module=Static_Docs&func=view&f=/setup.htm

Correct permissions so we can have the config.php created by the install script:

        # chown apache.apache /var/www/html/phplogcon/config.php
        # chmod 777 /var/www/html/phplogcon/config.php

Note: we'll change to a more secure setup after the install.php is complete.

Go to http://servername1.example.com.au/install/install.php

Click Submit on the first "Welcome to the install of phpLogCon etc" page.

Complete details for Connection Type, Host/IP, Port, etc.

Click "Install phpLogCon" which outputs:

        Checking users input... Done!
        Creating required tables... Done!
        Inserting values into tables... Done!
        Creating your config file (config.php)... Done!
        
        All processes have been done clearly!
        Congratulations! You've successfully installed phpLogCon!
        A file named 'config.php' is stored in the root directory of phpLogCon. In this file there are the whole information you
        have entered before! You can edit it to your needs if you want to.
        
        Move to 'index.php' in root directory to start working with phpLogCon!
        
        Don't forget to delete whole 'install/' directory!
        These files could be user for a DoS on your phpLogCon!
        
        After deleting the directory, you can go to index.

Secure the phpLogCon web environment

        # rm -fr /var/www/html/phplogcon/install/
        # chmod 644 /var/www/html/phplogcon/config.php

Login to phpLogCon

Click the index link, enter the username and password you created on the install screen.

Using phpMyAdmin, we can now look at the SystemEvents table in the Syslog database to show entries are actually being logged by counting the number of entries in the table.

We should also be able to now see these rsyslog entries in phpLogCon in the "Show Events" tab. If we don't, we need to turn off the UTC time setting as explained below.

Modify UTC time setting

As explained here:

http://www.phplogcon.org/Topic3.phtml

We need to modify the config.php to turn off the UTC time setting.

       # vi /var/www/html/phplogcon/config.php

modify:

       // Use UTC time, indicates if the records are stored in utc time
       define('_UTCtime', 1);

to:

       // Use UTC time, indicates if the records are stored in utc time
       define('_UTCtime', 0);

Now going back to the phplogon webpage, select to view entries for today, we should see our syslog entries.

Cleaning up and Migrating old configs

After installing the rsyslog RPM, I also had to perform some cleanups and migrations, for example:

        # rpm -Uvh /usr/src/redhat/RPMS/i386/rsyslog-1.18.0-1.i386.rpm
        Preparing...                ########################################### [100%]
           1:rsyslog                ########################################### [100%]
        warning: /etc/logrotate.d/syslog saved as /etc/logrotate.d/syslog.rpmsave

As we're now using /etc/logrotate.d/rsyslog instead of /etc/logrotate.d/syslog, I migrated my settings in /etc/logrotate.d/syslog to /etc/logrotate.d/rsyslog, then:

        # rm -f syslog.rpmsave

otherwise logrotate will produce a nightly error about that rpmsave file existing in that directory and being ignored.

Note: because the /etc/init.d/syslog stopstart script no longer exists, and I still have software which looks for specifically that stopstart script, I also created a link from /etc/init.d/syslog to /etc/init.d/rsyslog using:

        # cd /etc/init.d
        # ln -s rsyslog syslog

Logging just maillog entries

My purpose for using rsyslog was not to log anything of any level from any server. Doing that would cause the central MySQL database to grow so large and be cluttered with unnecessary information, causing support staff to waste time looking through items which have no relevance to them.

For my requirements, I'm only interested in the maillog entries, so I changed the entry in /etc/rsyslog.conf from:

        *.*       >servername1.example.com.au,Syslog,syslogwriter,LongEncryPtedPaSsWord

to:

        mail.*,*.emerg       >servername1.example.com.au,Syslog,syslogwriter,LongEncryPtedPaSsWord

(which logs all maillog and emerg) and restarted rsyslog with:

        # service rsyslog restart

Maintaining the MySQL database

Thankfully, the rsyslog developers are very responsive and appreciative of contributions from people like me. Because of such developers, it's projects like this I love working on and contributing to.

With 6 mail servers running and logging to a central MySQL server, after only 5 days I had well over 119,000 lines logged, which dramatically slows down the web access to the database. I don't have a requirement to keep maillog entries for longer than a week, so allowing the database too keep more than 7 days of entries is unnecessary.

In the rsyslog tarball within the "contrib" directory, exists a MySQL maintenance script called delete_mysql. This script doesn't in fact "delete" MySQL, but prunes the MySQL tables for you, so you should modify this script to add your database name, username, etc and use it via cron to prune and optimise your database periodically.

Changes in rsyslog v1.19.0 and above

From the rsyslog v1.19.0 release, MySQL is now a plugin so the rsyslog_v1.19.0 spec files (and above) now produce two RPM's, namely:

        /usr/src/redhat/RPMS/i386/rsyslog-1.19.0-1.i386.rpm
        /usr/src/redhat/RPMS/i386/rsyslog-mysql-1.19.0-1.i386.rpm

which you need to install consecutively. Obviously, the rsyslog-mysql RPM provides just the libraries for rsyslog with mysql enabled.

Once you've installed both RPM's, you need to modify the /etc/rsyslog.conf file to add the:

        $ModLoad MySQL

option at the top of the file or above the entry which sends rsyslog information to the MySQL server.

About the author

I've been using Linux at administrator and hosting (web and dedicated hosting) levels since 1994, from the early days of Yggdrasil, to Slackware, Debian, Red Hat and now to Scientific Linux[1], my preferred distribution.

I've worked for various companies over the years primarily as a Senior UNIX Administrator supporting medium to large UNIX environments, single server or clustered with Linux, Tru64 UNIX, Solaris and HP-UX.

Over the years I've also contributed my time and efforts into many Open Source projects including linuxha.net[2], Squid[3], Webmin[4], Virtualmin GPL[5], Virtualmin Pro[6], Webminstats[7] (and various other Webmin modules), smartmontools[8], clusterssh[9], BlockHosts[10], Brim[11], Kimai[12], DCC[13], OpenWebmail[14], mrepo[15], whichcd[16], rkhunter[17], csync2[18], drbdlinks[19], various perl modules, Scientific Linux[20], etc ... and now rsyslog[21] and phpLogCon[22].

I'm using rsyslog for various servers in a hosting environment which provides Web hosting, Webmail, Domain names and Mail scanning (Virus, Spam) solutions for various clients.

Michael Mansour.

Additional Links

This adds some links that are considered helpful, but are not yet merged into the main article content:

Personal tools
language