Overview
A rootkit is a group of software tools which an attacker can use to hide their tracks. A rootkit can also contain software which allows the attacker to get root access and steal or remove files on a system. Another goal for a rootkit is for the attacker to maintain access to the hijacked computer. Rootkits are written for many different operating systems however, this article will only talk about Linux rootkits.
Types of Rootkits
One type of rootkit is at the user level which is the simplest one and easiest to detect and remove. They can replace a user application with a modified program of their own. They are easier to detect because one can trust the kernel of the operating system. By scanning for programs which have been changed software like AIDE and Tripwire can detect this type of rootkit. Another type of rootkit is at the kernel level. These are harder to find and remove because one can't trust even the kernel on which the rootkit has been installed on. They have the ability to delete logs to hide the intruders tracks and even replace system calls. This type is usually installed as a Linux Kernel Module (LKM). Some examples of LKM rootkits are Afhrm and Synapsis.
Techniques Used in Rootkits
Using the Linux Kernel Module, a rootkit can modify the kernel's syscall table. Doing this the rootkit can replace a system call to point to a program of the rootkit. Another technique which a rootkit can use is to delete a log entry on the system so there will be no log entry of the attackers activities. Also, to hide the attackers tracks the rootkit can replace standard Unix programs such as ps to not show the processes which the rootkit is running.
Detecting and Removing Rootkits
The problem with detecting good rootkits are that you can't even trust the kernel and operating system in which the rootkit is installed on. So this makes it hard to detect them by installing detection software directly on the affected operating system. A better solution is to install a packet sniffer on an unaffected machine to look at the information being sent to and from the machine which might have a rootkit installed on it. Looking at the local log files will not always allows the system administrator to detect an attacker using a rootkit because the rootkit can delete the entries the attacker makes. Another way to detect rootkits are to boot from a live CD. This allows you to trust the kernel and the software running on the linux CD to investigate the files on the possibly affected computer for rootkits. Also there are programs which try to find rootkits locally like chkrootkit however this program depends on the local ps command to find them. As we know a rootkit can change the ps command to what it likes. Another problem with this approach is that the rootkit can detect and change the chkrootkit software. If the user finds the rootkit sometimes it is very hard to make sure that it has been removed. Most experts recommend that one should just reformat the system and start over. If using backups make sure that the backed up files don't contain harmful files. There is software which tries to remove rootkits called Rkdetector v2.0. Detecting and removing is so hard because they are designed to be hard to detect and remove.
Prevention and Monitoring
The best way to keep ones system secure and free of rootkits is to prevent them from being installed on ones system. One way for doing this is not allowing the attacker to have access to the administrative account. Without root access the attacker can't hide their tracks with a rootkit. To monitor a system a technique called file integrity checking is used to detect rootkits fast by looking at the machine for changes. The idea is to make a fingerprint of the machine right after a fresh install and after a newly installed program. A fingerprint is a cryptographic hash function which makes a hash that depends on every bit of data in a file. After this hash is made, by calculating and comparing the stored hash value with the current hash value, changes in the data can be detected. Also, there is Linux software which checks the integrity of files on a machine. One example of this type of software is Tripwire which uses a hash function and stores the information about the files in a password protected database. It will alert the user if the files which are being monitored have changed and then the user can see if a rootkit changed that file. However, the best way to prevent rootkits is by practicing smart security, for example, firewalls, good passwords, checking permissions etc.
Conclusion
Rootkits are a way attackers hide their tracks and keep access to the machines they control. The good rootkits are very hard to detect and remove. They can be running on ones computer and no one can even know they have been running. The rootkits which are unknown and uses LKM are one of the worst ones a Linux user can get. Practicing good security, for example, using SELinux which is used by EnGarde Secure Linux, is the best way to combat rootkits. For more information about rootkits and software please look at these references.
References
1. RK detector
http://www.rootkitdetector.com
2. Linux RootKits For Beginners - From Prevention to Removal
http://www.sans.org/reading_room/whitepapers/linux/901.php
3. Tripwire
http://www.tripwire.org
4. AIDE (Advanced Intrusion Detection Environment)
http://sourceforge.net/projects/aide
Thank you for posting these tools. | 
