AUTHORS BIOS

Eric Hines
As a 7 1/2 year member of the network security industry, Eric currently provides security consulting to many Fortune 500 to Fortune 50 Corporations across the US. Eric also gives keynote speakings on information security at numerous different E-Commerce and E-Banking seminars. Eric has deployed several VPN Extranets from British Columbia to the United States, as well as plays active roles in contributions to many security-related mailing lists. Eric enjoys creating custom attack signatures using packet analyzers for Intrusion Detection Systems such as SNORT, and holds strong interest in Intrusion Detection technology.

Fyodor Yarochkin
Works for TRG (http://www.relaygroup.com). Personal information available at: http://www.kalug.lug.net/fygrave/.

INTRODUCTION

Welcome to the first issue of a series of security-related HOWTO papers. In these documents both Fyodor and I aimed to give various tips, hints and tricks on various aspects of Internet security. Comments and suggestions are welcome and can be emailed to us directly at the contact information provided at the bottom of this paper.

We hope you find it useful and recommend hammering away at the README files for all utilities being referenced in this paper.

In this paper we make references to <SERVER> and <CLIENT> as headers. In this, SERVER, refers to nothing more then the machine that is configured to receive logs from remote machines (CLIENTS). CLIENT, referring to the machine that is configured to send its logs to the REMOTE LOG SERVER.

- Eric Hines

A. WHAT IS A REMOTE LOG SERVER

A remote log server is nothing more then a system preconfigured at install-time to provide hard drive space for other systems to log to. This system must be completely secured and locked down. No unencrypted remote access should be allowed, all RPC Daemons and other misc. services should be turned off as well. The only data allowed to the machine should be UDP/Port 514. We will be walking you through a step-by-step process that details how to configure, install, and deploy a remote log server. Utilizing some of the most renowned security experts across the globe for input, I've compiled a comprehensive, and easy to understand guide on ensuring this to be a successful launch.

B. SYSLOGD

Recompiling syslogd
The first step in turning on remote logging capabilities on your machine is recompiling syslogd to read a completely different syslog.conf file. This is for security reasons, which we will detail more later in this paper.

NOTES
This was an idea derived from the "To Build a Honeypot" whitepaper written by Lance Spitzner. Thanks for your contributions to the security community, Lance. The beautiful idea behind this strategy is to fool the intruder into thinking there is no remote logging setup on the machine. By leaving the old /etc/syslog.conf file in tact, also helps in tricking the unsuspecting cracker.

STEP 1: COMPILING OUR NEW VERSION

In our first step, we face somewhat of a problem in that syslogd ships with the operating system by default, so no source code is provided with it. You will need to download the source code from an outside party, such as Red Hat, or even Joey's FTP site (the creator of syslogd). As of this writing, Joey's ftp site is down so I have provided an alternative resource to download the needed file. Your distribution however, would be in violation of Open Source policies by not providing the source code to you. So check your particular OS vendor for the source code if you do not want to download it from any of the links specified in this paper.

SITE 1: ftp://ftp.infodrom.north.de/pub/people/joey/sysklogd/ (currently down as of this writing)

or

SITE 2: ftp://ftp.sourceforge.net/pub/mirrors/Red Hat/Red Hat/Red Hat-6.2/SRPMS/SRPMS/sysklogd-1.3.31-16.src.rpm

After 'rpm -iv -v sysklogd-1.3.31-16.src.rpm, the source code should have been installed to /usr/src/Red Hat/SOURCES/sysklogd-1.3-31 (that is, if you are running Red Hat)

STEP 2: CHANGING THE DEFAULT CONFIG FILE LOCATION

Here is the fun part. This is used to fool an intruder once he has compromised the system into thinking your log files are stored locally. We will be recompiling syslogd to read a completely separate configuration file, leaving the old default one in tact, just in case the intruder runs a 'cat' on the /etc/syslogd.conf file to check if remote logging has been enabled. By leaving the old configuration there proceeding our recompile of the new binary, ensures that we can fool the cracker into thinking otherwise.

Lets modify the default location of /etc/syslog.conf

  [ehines@myhost sysklogd-1.3-31]$ cd /usr/src/Red Hat/SOURCES/sysklogd-1.3-31
  [ehines@myhost sysklogd-1.3-31]$ vi syslogd.c

do a /grep for syslog.conf

This will bring us to

  #define _PATH_LOGCONF   "/etc/syslog.conf"

Change the above location value to something you prefer. I suggest something like /etc/.sys/CORE.conf or something ingenious like that.

Copy the new syslogd binary over the old (typically /sbin/syslogd)

BACKUP old version before doing this. As an admin, you should have already known that, right? :)

NOTES
The reason why we do not simply just start the old syslogd with the (-f conffile) switch is because if an attacker were to grab a 'ps' list of processes, the switch you used will show up, telling the intruder that we are specifying a completely different configuration file. Ultimately, defeating this purpose. I recommend leaving the old /etc/syslog.conf file in tact as it completely throws off the intruder. Also, keep local logs, as if the /var/log directory is empty, well, use your imagination on why.

I would have to make the suggestion to use a different filename, other then syslog.conf when specifying the new location. Understand that an intruder can also just do a find / -name "syslog.conf" and locate another copy of the configuration file.

STEP 3: MODIFY THE (REAL) SYSLOG CONFIG FILE

For example, to forward ALL messages to a remote host use the following syslog.conf entries:

  # Sample syslogd entry to forward all messages to a remote host.
  *.*            @hostname

  # To forward all kernel messages to a remote host the configuration file would be as
  follows:
  kern.*         @hostname

  # Remote Logging which includes local logging as well (for fault tolerance)
  kern.crit                    @finlandia
  kern.crit                    /dev/console

.. and so on.. repeat for all log files you wish to keep offsite. This includes third party applications.

C. THE CLIENT (REMOTE LOG SERVER)

With the client server, (or the machine accepting all of the remote logs), all you have to do is launch syslogd with the -r switch, specifying remote logging capabilities. Make sure that you remember to ADD: syslog 514/udp to the /etc/services file as well, on the server, or it will not listen to the specified port.

NOTES
If the network consists of different domains you don't have to complain about logging fully qualified names instead of simple hostnames. You may want to use the strip-domain feature -s of this server. You can tell the syslogd to strip off several domains other than the one the server is located in and only log simple hostnames.

ADD 'syslog 514/udp'to the /etc/services file.
If this entry is missing syslogd neither can receive remote messages nor send them

ADD '*.* @hostname' to the /etc/syslog.conf file.
To forward all Kernel logging messages to a remote host,

Make sure that if you do not specify an IP ADDRESS that you add the host to the /etc/hosts file to avoid DNS RESOLUTION errors.

If the remote host is located in the same domain as the host, syslogd is running on, only the simple hostname will be logged instead of the whole fqdn. (fully qualified domain name)

D. LOCKING DOWN THE REMOTE LOG SERVER

One of the most crucial steps in this project is to fully lock down the machine that is receiving all of the logs, or referred to as the remote log server.

1. Turn off (ALL SERVICES)

Remember, this is ONLY a log server, so all other services should NOT be turned on. Lets walk through that process right now.

  [root@myhost /etc]# cd /etc
  [root@myhost /etc]# vi inetd.conf

1a. TURN OFF ALL INETD SERVICES

You will want to put a '#' comment in front of EVERY service listed in this file. The inetd.conf file is IMHO one of the largest mistakes of the Linux operating system :). But the ability to disable the inetd services makes it all better :) You will want to comment out everything:

a. echo
b. discard
c. daytime
d. chargen
e. time
f. ftp
g. telnet
h. shell
i. login
j. exec
k. comsat
l. talk
m. ntalk
n. dtalk
o. pop-2
p. pop-3
q. imap
r. uucp
s. tftp
t. bootps
u. finger
v. cfinger
w. systat
x. netstat
y. auth
z. linuxconf
zz. swat

1b. DISABLE ALL RPC SERVICES

  [root@myhost /etc]# cd /etc/rc.d

Anything listed in the rc. directories that start with a capital "S" means that the service will start at boot time. In order to disable it from starting, you will want to 'mv to (a lowercase 's'). e.g. mv S11portmap s11portmap // This disables portmapper from starting at boot time.

You will want to do this with all unneeded services in this directory. These include: nfslock, apmd, netfs, identd, autofs, portmap, atd, pcmcia, I would even disable the inetd startup script as well, isdn, sendmail, gpm, httpd, vmware, xfs,linuxconf, local, You are basically (other then the obvious services) are going to want to disable all of these rc controlled services.You will want to go through all of the rc0.d-rc6.d directories looking for these same files to disable.

2. DISABLE ACCOUNTS

  [root@myhost /etc]# vi /etc/passwd

Go through your password file and remove practically every account that isn't being used. I would suggest downloading the tool, /bin/noshell (http://www.cerias.purdue.edu/coast/archive/data/categ50.html) and use this with all accounts in the password file that should not have remote login access. This is a very informative program that provides admins additional information on who is attempting to login with disabled accounts.

3. INSTALL SSH

Since the release of OpenSSH, giving the GNU opensource community free access to secure shell, telnet should now be obsolete in a security centric environment. GET AWAY from using telnet. With packet sniffers getting easier and easier to use (tools such as dsniff, 'www.dsniff.org') allows even the biggest nitwit to snarf packet data. So case-in-point, don't use telnet.

I bet you are so happy, because guess what? I'm going to walk you through the installation of OpenSSH! (ohh goody)..

3a. DOWNLOADING OPENSSH

Point your web browser to www.openssh.com, as of this writing they are on version 2.1.1 which has support for both SSH1 and SSH2 protocols. The release date was June 08, 2000. After downloading the source tarball, go ahead and do the following:

  [root@myhost ]# gzip -d openssh-2.1.1p2.tar.gz
  [root@myhost ]# tar -xvf openssh-2.1.1p2.tar

I can't realistically walk you through everything, so please save me some heartache, as well as yourself, and just read the INSTALL file before continuing. I have no idea what particular requirements your system has. I will however, walk you through my install.

First you will need working installations of both Zlib and OpenSSL (yes there is an OpenSSL project as well)

Zlib:
http://www.freesoftware.com/pub/infozip/zlib/

OpenSSL 0.9.5a or greater:
http://www.openssl.org/

Or, if you're even more lazy then the audience this paper was written for, (God forbid), then grab the RPM's from the link below. RPMs of OpenSSL are available at http://violet.ibs.com.au/openssh/files/support

Alternatively, Jim Knoble has written an excellent X11 passphrase requester. This is maintained separately at: http://www.ntrnet.net/~jmknoble/software/x11-ssh-askpass/index.html

If you wish to build the GNOME passphrase requester, you will need the GNOME libraries and headers.

GNOME:
http://www.gnome.org/

Also, grab GNUmake if you do not yet have it, OpenSSH has only been tested with this program.

  [root@myhost ]# ./configure && make && make install

This will install the binaries in /opt/{bin,lib,sbin}, but will place the config files in /etc/ssh

If you are using PAM, you will need to manually install a PAM control file as "/etc/pam.d/sshd" (or wherever your system prefers to keep them). A generic PAM configuration is included as "contrib/sshd.pam.generic", you may need to edit it before using it on your system. If you are using a recent version of Red Hat Linux, the config file in contrib/Red Hat/sshd.pam should be more useful. Failure to install a valid PAM file may result in an inability to use password authentication.

I like to run configure with '--without-pam' to disable PAM support. PAM is automatically detected and switched on if found. This is because I opt to use RSA Authentication instead.

  [root@myhost ]# tar -xvf zlib.tar
  [root@myhost ]# cd zlib-1.1.3
  [root@myhost zlib-1.1.3]# ./configure && make && make install

You shouldn't get any errors, if you do, sorry to do this, but refer to the README file. What I like to do when compiling something and something goes wrong, I just grep a keyword from the error on the provided README file(s) to see if there is anything specifically mentioned about my error. (Just a suggestion)

To install OpenSSL, you will need:

• Perl 5
• an ANSI C compiler
• a supported Unix operating system

  [root@myhost ]# tar -xvf openssl-0.9.5a.tar
  [root@myhost ]# ./config && make && make test && make install

Now lets build OpenSSH!

  [root@myhost ]# cd openssh-2.1.1p2
  [root@myhost ]# ./configure --without-PAM && make && make install

NOTES
Because I opt for RSA authentication, I run configure with the --without-PAM option as I am not a big fan of basic password authentication. With the advancements in Cryptography and more secure forms of authentication, PAM and other password-type authentication mechanisms IMHO will become obsolete within a few years

When you have successfully compiled and installed OpenSSH, you will receive the following information.

  Key generation complete.
  Your identification has been saved in /usr/local/etc/ssh_host_key.
  Your public key has been saved in /usr/local/etc/ssh_host_key.pub.
  The key fingerprint is:
  d5:74:83:d0:3f:c4:b4:d6:c5:39:1d:94:ee:9b:a8:61 root@soc1.priv.nuasis.com
  Generating DSA parameter and key.
  Your identification has been saved in /usr/local/etc/ssh_host_dsa_key.
  Your public key has been saved in /usr/local/etc/ssh_host_dsa_key.pub.
  The key fingerprint is:
  ed:58:65:b9:8b:fe:05:81:c2:8c:06:c9:cb:ac:bb:e6 root@soc1.priv.nuasis.com

3b. CONFIGURING OPENSSH

Even though the default installation of OpenSSH should work, I would take a quick stab at the ssh config files to see if you want to change anything.

Because we configured OpenSSH without PAM support, we want to enable RSA Authentication. Uncomment it in the /usr/local/etc/ssh_config file respectively (this is for the system-wide client configuration).

Then go ahead and edit the SSHD config file, /usr/local/etc/sshd_config

Lets go ahead and disable PermitRootLogin (do this by changing 'yes' to 'no' Go ahead and save the file and quit.

After installation, a host key and DSA key should have been created for you. So lets go ahead and move to the next step.

NOTES ON SSH v2
THE SSH PROTOCOL v2 SSH protocol version 2
In SSH v2, each host has a host-specific DSA key used to identify the host. However, when the daemon starts, it does not generate a server key. Forward security is provided through a Diffie-Hellman key agreement. This key agreement results in a shared session key. The rest of the session is encrypted using a symmetric cipher, currently Blowfish, 3DES or CAST128 in CBC mode or Arcfour. The client selects the encryption algorithm to use from those offered by the server. Additionally, session integrity is provided through a cryptographic message authentication code (hmac-sha1 or hmac-md5).

Protocol version 2 provides a public key based user authentication method (DSAAuthentication) and conventional password authentication.

AUTHORIZED_KEYS FILE FORMAT
The $HOME/.ssh/authorized_keys file lists the RSA keys that are permitted for RSA authentication in SSH protocols 1.3 and 1.5 Similarly, the $HOME/.ssh/authorized_keys2 file lists the DSA keys that are permitted for DSA authentication in SSH protocol 2.0. Each line of the file con­ tains one key (empty lines and lines starting with a `#' are ignored as comments). Each line consists of the following fields, separated by spaces: options, bits, exponent, modulus, comment. The options field is optional; its presence is determined by whether the line starts with a number or not (the option field never starts with a number). The bits, exponent, modulus and comment fields give the RSA key; the comment field is not used for anything (but may be convenient for the user to identify the key).

Note that lines in this file are usually several hundred bytes long (be­ cause of the size of the RSA key modules). You don't want to type them in; instead, copy the identity.pub file and edit it.

CREATING USER SPECIFIC KEY FILES
With each user that you want to have access to the SSH server for, run /usr/local/bin/ssh-keygen on the machine you are going to be ssh'ing from. After running ssh-keygen, goto $HOME/.ssh/ and cp identity.pub -> authorized_keys

KEY LOCATION OVERVIEW
The system you want to login to (the SSH Server) is where you store the $HOME/.ssh/authorized_keys file. This is the public key file for your user. The server will require this to decrypt your private key on your host machine. The system you are logging in FROM is where you store the $HOME/.ssh/identity file. This is the private key file you generated.

NOTES
For those of you wishing to use SSH v2 (be careful, it's a pretty new version. I'd stick with v1 to be stable and safe).

  [root@myhost /usr/local/bin]# ./ssh-keygen -d

This -d flag will create the DSA Public Keyfile needed for DSA Authentication. (When generating host keys, you can not specify a password)

Once you have generated your key files, go ahead and plop your sshd module file into the /etc/pam.d/ directory. You are probably wondering, jeese, will Eric provide me with a working sshd module? Well the answer is YES :) See below

  ---- snip -----
  #%PAM-1.0
  auth       required     /lib/security/pam_unix.so shadow nodelay
  auth       required     /lib/security/pam_nologin.so
  account    required     /lib/security/pam_unix.so
  password   required     /lib/security/pam_cracklib.so
  password   required     /lib/security/pam_unix.so shadow nullok use_authtok
  session    required     /lib/security/pam_unix.so
  session    required     /lib/security/pam_limits.so
  ----- snap -----

Remember, this is being used on my RH 6.2 box only. If it does not work, consult the generic copy provided with the SSH distribution.

OVERVIEW
Now you should have ALL services turned off and SSH as the only means of connecting remotely to the machine for administration purposes. The next step we want to do is setup that infamous host-based Firewall!

4. THE FIREWALL

Packet filtering doesn't really make sense on your logging box until you've got your tcp stack vulnerable to some fragmentation issue, since you have got all the services down anyway and you can control who is permitted to establish secure shell connections to the machine using ssh access control features.

It might be reasonable though to filter out syslog port (514/udp) but it doesn't really stop people from sending garbage to your syslog server, however it does make their job harder since it's udp which could be easily spoofed. This is at least until you have a multihomed machine where all logs come to internal interface so you can safely filter out everything that comes from the outside.

However here are basic recommended filtering rules for machine. For this purpose you can use either ipchains package if you're using Linux (last time I checked ipfilter didn't compile cleanly there) or try to use ipfilter package if you use something else (works perfectly on *BSD and Solaris machines at least).

For ipchains I recommend the following shell script.

  #!/bin/sh

  PATH=/usr/sbin:/sbin:/bin:/usr/sbin

  LOCAL_INTERFACE="192.168.1.1/32" # put your real IP address
  LOCAL_NETWORK="192.168.1.0/24"   # put your  local net IP address/mask here
  SSH_PERMITTED="192.168.1.2/32 192.168.2.3/32" # who allowed to ssh
  SYSLOG_PERMITTED="192.168.1.5/32 192.168.5.2/32" # who allowed to log syslog messages


  # deny everything
  ipchains -P input DENY
  ipchains -P output DENY
  ipchains -P forward DENY
  ipchains -F

  #permit ssh
  for ipaddr in $SSH_PERMITTED;
  do
    ipchains -A input -p tcp -s $ipaddr -d 0/0 22 -i $LOCAL_INTERFACE -j ACCEPT
  done

  # permit outgoing tcp
  ipchains -A output -p tcp -i $LOCAL_INTERFACE -j ACCEPT
  ipchains -A input -p tcp ! -y -i $LOCAL_INTERFACE -j ACCEPT

  # permit syslog
  for ipaddr in $SYSLOG_PERMITTED;
  do
    ipchains -A input -p udp -s $ipaddr -d $LOCAL_INTERFACE 514 -i $LOCAL_INTERFACE -j ACCEPT
  done

  # if you would like to log all the other connection attempts,
  # uncommend these...
  #ipchains -A input -p tcp -i $LOCAL_INTERFACE -l -j DENY
  #ipchains -A input -p udp -i $LOCAL_INTERFACE -l -j DENY
  #ipchains -A input -p icmp -i $LOCAL_INTERFACE -l -j DENY

For IP filter package, rules would look approximately like this:

  # close everything on local interface
  block in all on le0 from any to any

  # pass secureshell
  pass in on le0 proto tcp from 192.168.1.2/32 to 192.168.1.1/32 port = 22
  pass out on le0 proto tcp from 192.168.1.2/32 to 192.168.1.1/32 port = 22

  # or you can replace these two rules with
  #pass in on le0 proto tcp from 192.168.1.2/32 to 192.168.1.1/32 port = 22 keep state
  #pass SYSLOG
  pass in on le0 proto udp from 192.168.1.2/32 to 192.168.1.1/32 port = 514

5. LOG REPORTING

What good would a remote log server be if you weren't monitoring the log files. I would recommend several different programs.

• Logcheck - www.psionic.com
• Swatch - www.swatch.org

Below we've included a simple shell script (Courtesy of Mr. Bill Pennington) Which will archive the log files every day/hour/minute to an offsite location (using scp of course).

  #!/bin/bash
  #Simple script to rotate the log files on a daily basis
  #Bill Pennington 1/19/2000

  #Set the date variable
  date=`date +%m-%d-%Y`

  #Rename the messages file
  mv /var/log/messages /var/log/messages.$date

  #HUP the syslog daemon so it writes to a new file
  killall -HUP syslogd

  #Compress the file
  /bin/gzip /var/log/messages.$date

  #Rename the secure file
  mv /var/log/secure /var/log/secure.$date

  #HUP the syslog daemon so it writes to a new file
  killall -HUP syslogd

  #Compress the file
  /bin/gzip /var/log/secure.$date

  #Rename mail file
  mv /var/log/maillog /var/log/maillog.$date

  #HUP the syslog daemon so it writes to a new file
  killall -HUP syslogd

  #Compress the file
  /bin/gzip /var/log/maillog.$date

  #Then scp them somewhere

6. TIME

It is vital to ensure that the logserver has the correct time/date. Get xntpd install it, then "ntpdate timeservername" at least once a day, I do it twice a day. If you want to use your logs in any type of legal proceeding you are going to need accurate time. You should be doing this on all your servers of course.

7. OTHER SYSLOG DEVICES

Their are many other devices including CISCO that can be setup to log to syslog.

Having all your cool *nix boxes logging to a central log server is way cool, but what about those other boxes on your network? To get a complete view of your network and servers, setup all your devices to log to syslog. Depending on the number of devices you have, you might want to dedicate a box for each type of device or server. i.e. unix log server, netgear log server, NT log server...

Cisco
All Cisco devices support logging to syslog. The commands to set your Cisco device to log to log messages to syslog vary from device to device. Some examples are provided here but you should check your Cisco documentation for more complete instructions.

Routers

In config mode...

logging <server ip address or name>
logging facility <facility> defaults to LOCAL7

Pix Firewall

In config mode... logging host <ip address of syslog server> logging facility <facility> defaults to LOCAL4 logging trap <level> from emergencies to debug , be carefull with debug you will get a ton of traffic!

Switches

In config mode...

set logging server enable
set logging server <ip address of log server>
set logging session enable
set logging level all 1 default

Again you will want to look at your Cisco docs or Cisco's web site for complete details but the above examples should get you started.

Windows NT
You can make NT servers forward events to central syslog servers as well. There are several shareware/freeware packages available to accomplish this. Search http://www.bhs.com for syslog to find one that is right for you.

D. RESOURCES

• Swatch - http://www.stanford.edu/~atkins/swatch/
• OpenSSH - http://www.openssh.com
• LogCheck - http://www.psionic.com/abacus/logcheck/
• xntp and list of public NTP servers - http://www.eecis.udel.edu/~ntp/
• Cisco - http://www.cisco.com
• Windows NT software - http://www.bhs.com
• Robert Graham's sniffing FAQ (hints to build `receive-only device' http://www.robertgraham.com/pubs/sniffing-faq.html
• IP Filter - TCP/IP filtering package: http://coombs.anu.edu.au/~avalon/

E. CONTRIBUTIONS

Lance Spitzner (Renowned security whitepaper author of the 'Know Your Enemy' series. Thank you for all your support Lance. http://www.enteract.com/~lspitz

Martin Roesch (Creator of the popular Intrusion Detection System (SNORT)). Had a blast with you at Defcon! Hope to see you next year. http://www.snort.org

Bill Pennington (Thanks for all the help with SNORT and your contributions to this paper!) Good luck with everything!

Frank Heidt (@Stake Security Team - Enjoyed the cigar Frank! Hope to see you at Defcon next year as well. Sorry I didn't end up implementing your syslog design into this paper. I didn't think I could explain as perfectly as you could;))

Fyodor Yarochkin (This paper wouldn't have been possible without you bud! Thanks!)

F. SHOUTS

Lamagra, Safety, #RootHat, zen-parse, lockdown, Crimson, Invader, LoA, Fyodor, Faisal Jawdat, Art Stine, Vince Pereira, Ted Barban, Malverian, Frank Heidt, Bill Pennington, Marty, Lance.. and all the rest. "TIMMMMMY!"

G. CONTACT INFORMATION

Eric Hines
ehines@northernwatch.com
erichines@mail.com
http://profiles.yahoo.com/seattlerainstorm

Write Comment
Please keep the topic of messages relevant to the subject of the article. Personal verbal attacks will be deleted. Please don't use comments to plug your web site.. Such material will be removed.
Name:
Title:
Comment:
Code:*

Powered by AkoComment!