Patching and upgrading software requires more than running a few commands. Having a patch recovery plan, communicating with developers on that server, and knowing who to contact in case of a botched patch job is critical.

I wonder what goes through Jay. F.'s head when I send another patch update with a few dozen servers to patch. Patch Management can be a headache, especially in a large network environment. It can also be disastrous if someone doesn't read the documentation that comes with patches or types the wrong command to upgrade a software package. Consequently, knowing how to back out of a botched patch job is just as important as knowing how to apply the patch.

Don't apply multiple patches at once. Apply patches in increments, it makes recovering from a problem with one patch a whole lot easier. Patching will require you to know the functionality of services running and the current security settings. Some patches may disable security settings, overwrite a configuration file or may start services that were shut down or weren't even previously enabled. Backup configuration files for a software package before applying a patch to it. Also, run the following commands to see if something was started or disabled after the patch was applied.

/bin/ps -ef > before-patches-ps.txt

and

/bin/netstat -an > before-patches-netstat.txt.

Then after the patches run:

/bin/ps -ef > after-patches-ps.txt

and

/bin/netstat -an > after-patches-netstat.txt

Run diff or sdiff on the those files to see if anything has changed.

/usr/bin/sdiff before-patches-ps.txt after-patches-ps.txt

Also, check your file-integrity program to see what changes were made to the system. File-integrity programs have more uses than just looking for intruders.

Many vendors have a patch management program that will check your system against the most recent software versions available and download or send an email with the latest available versions. It is recommended to only allow these automated programs to alert you when there are new packages available. Others have mailing list and some vendors publish updates on Bugtraq. Before you patch your systems:

  1. Know who to contact before applying a patch, in case something goes wrong.
  2. Write down the current version of the software to be patched.
  3. Have that version on standby in case something goes wrong.
  4. Write down the version to be updated.
  5. Read the documentation about the patch before applying it.
  6. Check the website's mailing lists, if any, to see if others have had a problem with that patch.
  7. Take a deep breath and apply it.

Before patching, talk to the programmers and others who use the system about what is going to be done. If patching will break an existing service, for sure, then monitor that service more closely and document (GASP!! THE "D" WORD) why the service can't be patched and give a copy to your supervisor. Last, but not least, email the software's developers and explain that the patch didn't work properly.

Case example

K.D. was upgrading packages on his Linux server and he ran the commands:

/bin/rpm -Uhv *.rpm

and

/sbin/reboot

The server rebooted but his Linux installation didn't boot. His system was hosed and had to reinstall the whole system. After reinstalling, applying updates, and restoring from backups, a half a day had gone by and this was a critical server.

The error in K.D.'s case was that the latest kernel was within the rpms that he upgraded. He didn't know that the kernel should be installed with the command:

/bin/rpm -ihv kernel-version.rpm

instead of:

/bin/rpm -Uvh kernel-version.rpm

The switches, "ivh" installs the new kernel and modules and keeps the original kernel and modules. The switches "Uhv" removes the current kernel and modules and installs the new kernel and modules. He also didn't edit the boot loader configuration file to point to the new kernel.

K.D. had backups but no patch management recovery plan. K.D. was in an organization where he could have had a plan written up and reviewed by others. If K.D. had a recovery plan and communicated with the right people he could have gone into rescue mode, edited the boot loader configuration file, and rebooted with the new kernel with minimal down time.

Managers, require your admins to provide you a written patch management plan. The plan should include:

  1. Sites, mailing lists, and other sources to monitor for each service running on the servers they maintain.
  2. The primary and backup person to apply patches and their contact information.
  3. Contact information for any developers, programmers, or other admins running mission critical programs on that system.
  4. What is to be done if a patch breaks soemthing eg. what is the backup procedure and who to contact.
  5. What system to test patches on. Some organizations may not have the luxury of having test systems. That shouldn't be a problem because there are many operating system emulators available An investment in VMWare may be worth pennies compared to a day of lost revenue because a single patch broke your companies credit-card processing system. There is also the excellent GPL'd program User-mode Linux which emulates a Linux environment. With either program you can setup your production servers same software configuration and apply patches and see if it causes any problems.

Having a patch management disaster recovery plan, good communication, and cooperation will make patch management a seamless process. Also, Managers we need your support.

Duane Dunston is a Computer Security Analyst at STG Inc. for the National Climatic Data Center in Asheville, NC. He received his B.A. and M.S. degrees from Pfeiff er University and he has his GSEC certification from SANS. He hangs out at Old Europe Cafe, Early Girl's eatery, Annton y's, and any place with good tea and hot chocolate.

Duane has been working in security for 5 years and wishes he had the funding for a "Basic Security Tour" so he could provide the world with hands-on training on how to implement the security recommendations from the Sans Top 20 List of the most common vulnerabilities. He knows that applying these recommendations to any network can minimize the most com mon types of attacks. Not only does he enjoy his work in computer security, he also likes to get involved in its ever-g rowing technologies. Duane says, "Security is one of those jobs where you have to stay abreast of new technologies and new ways that attackers are compromising computer systems. Security keeps evolving and the industry has to keep up w ith it, that is why we need well-trained, evolving security professionals supportive managers to help us with this ongo ing process".