Centralized monitoring is critical today with the large number of servers that are deployed in many organizations. Reading reports from individual servers can be quite cumbersome and time-consuming. Programs are needed that will provide accurate and secure information about the state and health of servers in a timely manner in a centralized location, whether with one server or multiple servers. One of the most critical needs is a program that will alert an administrator when files have been changed on a system whether intentionally or as the result of a security compromise.

Many programs exist that provide from very basic file-integrity to very specific file-integrity. For example, some file-integrity checks look for any file that has changed within a directory and others can ignore if a file has been accessed or grown in size but alert if the permissions have changed. The level of security needed for file-integrity can depend on a number of factors. How many servers are in an organization, the time an administrator has to spend looking at file-integrity reports, the level of training for the administrator, the cost of a good file-integrity program, the sensitivity of data stored on a server, etc.. There are some superb commercial applications that provide fine-grain control of file-integrity configurations. Of course, it can cost quite a bit of money for these programs but if the needs fit an organization this may be their only solution. There are some great free programs that can rival many of the commercial applications. However, some require extra configuration with other third-party software to have logs sent to a central location securely, for example. Some organizations don't have the administrators who have the time to create custom programs to add extra functionality to the software. There exists software that will run as a daemon on all servers to be monitored, including the centralized monitoring server, send reports over a secure channel, by default, provide access control to determine whether that hosts is allowed to send alerts to the daemon, deploy the client on remote hosts over SSH, runs on Windows, Solaris, Linux, AIX, and other platforms, with tons of other features that is available under the GPL license, Samhain.

Samhain (pronounced "Sowen". "sow" as in a female pig) is a file-integrity program that provides high-level configuration for determining what files and/or directories have been changed, deleted, or added. It can also keep a record of files that have the suid or sgid bit set, as well. For Linux, it can send an alert if a new kernel-module has been loaded. Samhain provides a secure, centralized logging facility by having the client(s), that perform the integrity checks, on servers, send encrypted reports to a remote server. Samhain provides a ton of extra features that are very well documented in the manual that comes along with the distribution. Of course, this series of articles will give you a real-world working example of how to deploy Samhain.

Installing Samhain

The central logging server will be called pango.example.com.

To install Samhain download the samhain tarball, verify the pgp signatures, and extract it.

Linux:

Solaris & AIX:

#############################################################

SERVER INSTALL -- ONLY NEEDED FOR THE CENTRAL FILE-INTEGRITY SERVER
If you are using RPMS then you will need to download the mysql-devel package in order to compile Samhain with MySQL support

FILE INTEGRITY SERVER...IN THIS CASE, pango.example.com.

Samhain server install configured to log to a mysql database:

NOTE: the "--with-libs=-L/usr/lib/mysql/" and "--with-cflags=-I/usr/include/mysql" are specified, explicitly, because on Red Hat the configure script doesn't properly locate the "libmysqlclient.a" library or the "mysql.h" header file.

NOTE:On AIX 4.3, the Makefile had to be modified and have the "-O3" optimizations flags removed. These appear to specific to gcc. If gcc is installed on the server then it may work with the modifying the Makefile. This only needs to be done if an optimize error occurs during the "make " process.

The "make install" will create a program called yule in /usr/local/bin. Yule is the server that listens on pango.example.com for incoming integrity reports. Its configuration file is /etc/yulerc, by default.

The "DatabaseSeverity" directive will have to be specified in the /etc/yulerc file on pango.example.com.

Be sure that the yulerc file is configured to be readable and writeable only by root.

The next article will explain how to setup the yulerc file so that the database password (actually the entire yulerc file) and the Samhain daemon can be virtually hidden.

Samhain currently supports MySQL and PostGRESQL. The nice thing about logging to a database is that you can then use PHP, Java, PERL or some other language to extract the data and view it over the web or create your own purdy reports. Samhain has a beta program called Beltrane that will do this work for you and provide an interface over the web to view the alerts and logs for clients.

END SERVER INSTALL

#############################################################

Client install: (See NOTE: for AIX install)

This will add the samhain configuration file, samhainrc, to /etc and install the "samhain" executable in "/usr/local/sbin". That executable will be overwritten with the next step. Create a unique 16-digit number that will be added to pango.example.com to validate sending logs to the Samhain server daemon. All logs will be encrypted and sent to pango.example.com in the XML format, which will allow logging to the mysql server.

While in the root directory of the samhain source distribution run the command:

You have to choose a random 16-digit number to pass to the "samhain_setpwd" command. This will create a new executable for samhain in the current directory called, samhain.pudding. The "pudding" is really an arbitrary name, actually it is my sister's nickname. Move that one to the "/usr/local/sbin/" directory which will overwrite the default samhain executable created during the "make install" process.

Copy the 16-digit number to a sheet of paper or open a console on pango.example.com. This number will be used to create the unique checksum to allow this server to authenticate and send logs to pango.example.com. On pango.example.com...run the command:

For example, in the example above. Run the command:

The above command creates this checksum:

append the output of the "yule -P" command to the end of the /etc/samhainrc file, under the [Clients] section and change "HOSTNAME" to the name of the server that the number was created on.

Configuring the client:

The /etc/samhainrc file is heavily documented and provides various levels of file-integrity. It can be configured to check files and directories, that always change, ignore access times, size, check for new files, deleted files, etc.. It can also check for new suid and sgid permissions, it will also send alerts if a file or directory has a uid or gid that doesn't exist on the system.

Below are some specific cofigurations added outside of what already exists in the samhainrc file. The default configuration file is very well documented with some examples. Below are some configurations that were added for my setup.

Don't send duplicate reports for a modified file:

After configuring the Samhain /etc/samhainrc file, create the database. This may take some time, depending on what files and directories it has to create a checksums for and the suid files it has to check. Also, the priority specified in the configuration file can impact how fast the database is created.

After the database is created and the steps have been followed above to create a unique key for the host the samhain client is on, start the samhain client

This will run samhain as a daemon (-D) and send alerts every 120 seconds (-e 120) to pango.example.com. Use 120 seconds until you are comfortable with your setup then you can set the time between checks longer. Note that in the configuration file the "SetNiceLevel" is set to 19, which means the file check will run quite slow. This will keep the samhain daemon from using up a lot of CPU but it will cause the reports to be delayed when sent to pango.example.com. You may want to set the "SetNiceLevel" to something lower until you have Samhain configured correctly. Be sure to check the /var/log/yule_log file on pango.example.com for any problems with the hosts sending alerts. If Samhain, on pango.example.com, doesn't have a valid key for the hosts then it will not accept any alerts from it. A message will be echoed to /var/log/yule_log alerting that the host is invalid.

To start the server on pango.example.com, in this case:

This will run the yule server in the background. Again, check the /var/log/yule_log file for an error messages with yule starting up or check it to be sure that the yule server started properly. If you started the server after the client, wait a few minutes to see the connection attempts by the clients:

When the connection attempts start and are successful, there will be a lot of information echoed to the /var/log/yule_log file. This information can be tuned up or down depending on how much information you want. There is also an html file in the /var/lib/samhain directory called "yule.html". This file is updated every 120 seconds, by default. It reads the /etc/samhainrc file for the hosts that are allowed to connect to the yule server and gives their status. It will display whether a client is connected or whether a policy has been sent from that client, etc.. You can create a symbolic link from that location to the web server directory on pango.example.com and view it anytime. Be sure that you read and understand the security risks involved with creating symbolic links in a web directory. For any monitoring server, I like to use SSH tunneling to access it. Generally, no services are available directly from the public interface. For example, the web server can be forced to listen on the local interface for incoming http requests in the httpd.conf file with the Listen directive set to:

In order for anyone to connect to the server they will have to be on pango.example.com from the console or they can create an SSH tunnel to access the web server.

Conlusion

It is hoped that this article provides a good introduction with some of the capabilities of Samhain, how to install it, and set up a centralized file-integrity server. Samhain comes with a detailed manual in the source distribution. The next series of articles will explain other features of Samhain especially how to deploy Samhain, quickly, to multiple servers, hiding Samhain, using the yule server for a remote syslog server, how to setup Beltrane, and sample php and servlet scripts for extracting data from the Samhain log file or database to create customized web-based reports.

Special thanks

First and foremost, thanks to the Linuxsecurity.com team for their continued support with publishing my articles. Especially, Dave Wreski since I never send him an article on time because I always catch something that isn't right about it at the last minute. Thanks for your patients Dave. ;-)

Some LinuxSecurity list members provided a lot of help with proofreading this article and giving suggestions and comments: Emily Ratliff (many thanks!!), Bernard Hoffmann, Matt Hemingway, and Andy Grimm.

Thanks to Bone, Chris, Cris, Barium Spring Home for Children ("The Foundation of Duane's Path to Liberation"), Charla, Chrissy, Mr. David, Bob, Donna, CFCC, Pfeiffer University, Leslie, Adam, STG, NCDC, Lauren, Jason, and mutsman for their continued support for all that I do.

About the Author

Duane Dunston is a Computer Security Analyst at STG Inc. for the National Climatic Data Center in Asheville, NC. He received his B.A. and M.S. degrees from Pfeiffer University and he has his GSEC certification from SANS. He writes poetry, just started photography, and hangs out at Old Europe Cafe, Early Girl's eatery (tell'em Duane sent you), and still wakes up every morning ready to go to work. If anybody works at NBC who reads this, tell Ann Curry he says, "Hello", again.