Chat Transcript: Real World Linux Security with Bob Toxen

<bdthomas> ------------ BEGIN -------------

<bdthomas> Hello everyone, welcome to the 2nd LinuxSecurity.com chat event. I would like to welcome today's guest, Bob Toxen.

<bdthomas> Here is Bob's Biography:

<BobToxen> Thanks.

<bdthomas> BOB TOXEN has 28 years of UNIX/Linux experience. One of the 162 recognized developers of Berkeley UNIX, he learned about security as a student at UC Berkeley, where he cracked several of the original UNIX systems there. He is now President of Fly-By-Day Consulting, specializing in Linux and network security, firewalls, VPNs, 24x7 network monitoring, response, and administration for clients worldwide.

<bdthomas> Here is the URL to Fly-By-Day Consulting, Inc:

<bdthomas> Linux Security Services, Linux Firewall, Linux Anti Virus Filter, NetworkSecurity

<bdthomas> It is based out of Atlanta, Georgia (USA)

<bdthomas> Bob has authored several Books

<bdthomas> Real World Linux Security (November 2000) and Real World Linux Security: Intrusion Protection, Detection, and Recovery (January 2003)

<bdthomas> You can find them here:

<bdthomas> BestBookBuys.com

<bdthomas> Bob, if there are any specific places you want people to go to buy your Books, please let us know.

<bdthomas> Ok Bob, are you ready to begin the questions?

<BobToxen> Barnes & Noble has treated me especially well but I know that they don't have the best prices. Borders stocks my book in most locations.

<BobToxen> I'm ready. Hack away.

<bdthomas> I've noticed the Barnes & Noble in Louisville, KY keeps them both stocked well.

<BobToxen> Thanks.

<bdthomas> Ok first question:

<bdthomas> Can you offer a brief description of your background.

<bdthomas> (in addition to bio)

<BobToxen> I have about 30 years of intense C programming, including Unix and Linux kernel hacking and Assembly programming, which is a good start for understanding security. I also have some Accounting background which also helps for that way of thinking...

<BobToxen> I've been doing solely Linux and network security since January of 2000.

<bdthomas> Question 2:

<bdthomas> Tell us a little bit about your books, Real World Linux Security. Why did you write them. Who is the intended audience, and what overall message are you trying to portray?

<BobToxen> I wrote them mainly to help Linux. While Linux is capable of excellent security -- far better than Windows despite what those lying bleeps at M$ say -- there were no good books or other easy way for SysAdmins to learn how to secure their Linux systems and mixed-OS networks...

<bdthomas> Lying Bleeps .... exactly. :)

<bdthomas> We've all seen the ads comparing Linux TOC with Microsoft. What is your

<bdthomas> viewpoint on this? What experiences have you had as a Linux security

<bdthomas> consultant with TOC?

<BobToxen> My audience was SysAdmins, both experienced and newbies for both large and small networks. One of my technical reviewers deliberately was a Linux newbie. If she could not understand a section, it was rewritten, sometimes repeatedly until she did. I had several other technical reviewers who were experienced SysAdmins, and two security experts to ensure I tell the whole story.

<bdthomas> That entire process (book editing) sounds very interesting.

<bdthomas> -- If YOU have questions for Bob, please join #questions and post them. --

<BobToxen> Follow the money. Microsoft is, IMHO, far less reliable, crashes more, requires faster hardware to do the same job, and has outrageous license fees. How could it possibly have a lower TOC. Let's get back to Linux.

<bdthomas> Bob, to help with continuity .. please end your responses with ** So I'll know when you're finished.

<BobToxen> While writing the manuscript I told myself, that as soon as the manuscript was done, the rest was easy and quick. I suspect that I was lying to myself then. For the second book, I KNEW I was lying but that's how I kept going. The editing process is timeconsuming but critical to a good book.

<BobToxen> **

<bdthomas> Describe what it was like writing your first Book. What kind of hours did you spend? How long did it take?

<BobToxen> If you have to ask...As I was about to dive into the process I was in a large dinner and made an offhand comment that I was about to start writing a book and someone asked, "Will you be taking a leave of absence from your business?" That was the first hint of trouble...

<BobToxen> I ended up doing nothing else except sleeping 5-8 hours every 24-36 hours besides working on the first book for about 6 months. The rest of my world stopped. I kept my then girlfriend busy visiting her mother, friends, camping with friends, etc., so I could write. I also learned a heck of a lot about security, far, FAR more than I thought there was to learn. **

<bdthomas> It definitely takes a lot of commitment. Great story.

<bdthomas> Ok, now shifting gears.

<bdthomas> Tell us about the 7 deadly sins of Linux Security, What is it? How did you come up with it?

<BobToxen> Security experts say don't do this, that, and a hundred other things for security. That's too much for one to worry about. I decided to come up with the most critical -- the ones where, if you committed any of them, you almost certainly were going to get hacked. Calling them the 7 deadly sins" got people's attention. I have a 1-2 hour talk on it and it's my most popular talk, by far. I came up with it by studying compromised systems and determining wh

<BobToxen> what mistake, if avoided, would have prevented it.

<BobToxen> The 7 sins are things like not keeping up-to-date w.r.t. patches, weak passwords, etc. **

<BobToxen> Let's not forget open network ports and procrastination. **

<bdthomas> Yes, I think its popular because people don't have time to secure every aspect. They'd rather focus on the major problems.

<bdthomas> Next question:

<bdthomas> What are some of the linux security tools that you depend on everyday?

<BobToxen> Nobody is going to do everything right. In any case, one has to start somewhere, and the 7 deadly sins they should start on right now.

<BobToxen> My favorite tools are nmap, netstat, and find...

<BobToxen> I use nmap to analyze a client's network as the first pass, netstat to analyze an individual system for open ports...

<BobToxen> I use find to find files with bad permissions, such as

<BobToxen> being world writable, having set-UID or set-GID when they should not. **

<bdthomas> Great, thanks for sharing. That's valuable because it shows Linux newbies that one does not have to use dozens of tools to begin securing their system.

<bdthomas> When installing a Linux system (from a CD or the Net) what are some of the first steps you take to make sure it doesn't get compromised.

<BobToxen> First, don't connect it to even a LAN until it is hardened. Then start with Chapter 2 of my book, closing unneeded network ports -- with NFS and portmap first, turning off named (DNS) and Apache unless they are needed. Then fix file permissions. Even the latest versions of most distributions still have these stupid problems. Then download and install the latest security patches. **

<bdthomas> Again, great advice.

<bdthomas> Next question:

<BobToxen> Also, pick STRONG passwords, consisting of at least three words and several non-alphanumerics. Don't even waste your time with using ones for els and zeros for ohs. The hackers will crack anything less and will try those substutitions. **

<bdthomas> What's your view on the current state of Linux Security? What are some of its biggest problems? How has it improved over the years? (which improvements are best)

<BobToxen> Linux Security has gotten much better...

<BobToxen> The single biggest improvement is the automatic patching tools, such as Red Hat's up2date. This is because people will only do this manually every few months, if at all. Allowing up2date, for example to do it once or twice a day will shrink the window of vulnerability tremendously. I do note that up2date is not the best written program I've used. It is finicky and sometimes requires lots of fiddling to keep working...

<BobToxen> Most distributions HAVE improved not enabling as many unnecessary processes, have less files with the wrong permissions, and several auditing projects have fixed LOTS of buffer overflow vulnerabilities even before they could be exploited...

<BobToxen> On the down side the average desktop or Laptop system has so much "stuff" running on it from so many places that this increases the risks. Few people run Linux-suitable mail virus filtering and have all sorts of attachment types that will fire up automatically, including .doc. That scares me. I don't even enable Java in my Opera browser. **

<bdthomas> What improvements do you see coming to Linux's security in the near future? What would YOU LIKE to see to Linux security in the long-term?

<BobToxen> I think that the near term improvements will be more of the same. I think, too, that people are more security-aware. I'm finding that now even many non-computer types know that Linux is more secure than Windows and are willing to go to Linux. That's a GREAT improvement. More people now are worrying about security before they get hacked, even if it is a home system. Almost everyone uses some sort of firewall or virus filter now...

<BobToxen> Long term, again, I think more of the same in that there will be code improved in quality to reduce the risks and more hardened systems. I see NO improvement from the gov't or ISPs trying to filter out attacks or discourage hacking. I think that many hackers are just trying to teach people not to use junk systems that have no security...

<BobToxen> When some of my non-computer types say things like "let's have the death penalty for hackers," I ask what about billionaires who ship systems that they know are defective but who don't care. **

<bdthomas> Yeah, the death penalty is a little harsh ..

<bdthomas> I agree though, with everything.

<bdthomas> Next question:

<bdthomas> Have you been involved in penetration testing? If so, what tools do you normally use?

<bdthomas> Do you follow any specific methodology such as OSSTMM?

<BobToxen> I don't find penetration testing (PenTesting) to be useful for anything besides convincing executives who don't think that they have a problem that they do. A PenTest will show that there is at least one vulnerability but does not address the 10 or 100 that there might be. Properly hardening a network, patching, closing open ports, etc. will allow one to be satisfied that they have good security...

<BobToxen> My favorite example is a company in California that hired me to harden their network to the maximum extent possible and was willing to pay for it. I did so and then they scheduled an 8 hour PenTest. My adaptive firewall locked out the 5 IP addresses that the Pen Team was using to break in in only half an hour. The PenTest was over and they didn't find anything...

<BobToxen> Which was the better expenditure of funds, my hardening or the PenTesting? They then asked me to unlock the IPs and turn off the adaptive capability. They then spent the rest of the 8 hours and didn't find anything of consequence...

<BobToxen> I don't spend a lot of time PenTesting. Instead, I'll check versions of software and use nmap. I then can say, "Weeeell, I probably *could* break in by hitting this IIS server here, or that NFS server there, etc. There's not really a point to actually doing so, that I can see. **

<bdthomas> Yes, great points. Spend time where it will actually help.

<bdthomas> As a consultant, have you ever been involved in cleaning up after an attack, or forensic investigation? If so, how would you describe the experience?

<BobToxen> I've had my share of both. Like a dentist, I'd rather clean teeth than drill them...

<BobToxen> UNlike those who say of a compromised system, "just re-install from backup," for a production e-commerce system that can be rather inconvenient. I've developed a technique to analyze what has changed since the last backup and determine which of those changes are legitimate and which are Trojans. I then remove the Trojans and they are back up. It typically takes a day to do and is VERY effective...

<BobToxen> It basically involves using tar, if that is the format of the backup, and running tar in its comparative mode to identify files whose data or permissions differ between disk and tape. Of course, you must boot the system from a trusted CD or floppy since the kernel and system libraries on disk could be compromised...

<BobToxen> I've done some forensic work, including a case where child pornography was strongly suspected to be on a gov't system. My involvement started when I got a phone call at 8am from the SysAdmin, who explained that the local and state police walked in -- and clearly looking to take someone away in handcuffs...

<BobToxen> This was a case where I had hardened the server and a SysAdmin then went in and made changes that unhardened it. Subsequently, someone took advantage of one of the newly introducted vulnerabilities to traffic in the forbidden material, unbeknown to us. The SysAdmins then had me re-harden the system and THEN the previous trafficing was discovered. Because the problem already was fixed, no names ended up in the paper and nobody was fired...

<BobToxen> That's a fine answer to "Why should I worry about security, I have nothing interesting." The server in question was just a caching web proxy. **

<bdthomas> Wow, such an interesting story.

<BobToxen> It's never dull and I never know what my next client will need! **

<bdthomas> Also, I'm sure that many of our readers would be interested in your 'tar in comparative mode' technique. Have you written anything about it? Are there any places online that those interested can reference? Where would be a good starting point if someone wanted to do something similar?

<BobToxen> Again, it's described in Part IV of my book, part of the 50,000 or so words on preparing for a break-in and what to do when one suspects that it has happened. Other books just say "recover from backup". Doing "man tar" will show the flag in GNU tar. You then need to run "find" to find those files that tar did not compare. Those files are new files and should be very suspect. I don't know of any online descriptions of the techniques. Next question. *

<bdthomas> Thanks.

<bdthomas> Do you believe the open source nature of Linux provides a superior

<bdthomas> vehicle to making security vulnerabilities easier to spot and fix? What is your feeling about full-disclosure vs. 'responsible' disclosure.

<BobToxen> Great question!

<BobToxen> Open Source prevents a dishonest or lazy vendor from hiding behind "It's proprietary so we can't show you but we really, really promise that it is secure...

<BobToxen> Open Source allows hundreds of white hat eyes to look for and fix problems. Anyone who does not trust the software's creator can hire their own security experts to do such a review too...

<BobToxen> Having worked for several Unix hardware vendors in the past, including Silicon Graphics, I can assure people that in the proprietary world there is typically 1-3 people who ever look at a piece of code and they do not have time to worry about security. It's get it working well enough to meet shipping schedules...

<BobToxen> For those who think that keeping the code "secret" will help, well, the hackers simply disassemble it or break the vendor's security and get the source. There is STRONG evidence that this has happened with Microsoft code. It certainly has happened with Unix code where almost anyone involved has Unix source code in their garage. I'll neither confirm or deny that I do. ;^) ...

<BobToxen> The term "responsible disclosure" is used mostly by dishonest vendors who are trying to get laws passed to prevent full disclosure so that people cannot even talk about their vulnerabilities so that they don't have to worry about being caught or spending the money to fix them...

<BobToxen> It has been well established that only the threat of full disclosure will get many vendors, both M'soft and some Unix vendors to fix their code promptly...

<BobToxen> I DO agree that the discoverer first should notify the vendor or maintainer and give them a week or so to fix and distribute the vulnerability. If the programmers have to pull allnighters, that'll encourage them to be more careful next time. Putting a "gag" on disclosure as some have done certainly is a violation of what's left of Constitutional protection here in the U.S. **

<bdthomas> Great perspective.

<bdthomas> Do you have time for two more serious questions, and one more not-so-serious?

<BobToxen> I have as much time as people have questions. **

<bdthomas> Ok, thanks..

<bdthomas> Next Question:

<bdthomas> What techniques do you recommend for managing large log files? Also, what do you recommend for managing system event notifications, alerts, etc.?

<BobToxen> Log files! An important topic...

<BobToxen> I have taken Logcheck, an Open Source product on most Linux Distributions, that analyzes log files and substantially enhanced it...

<BobToxen> I find it far supperior to LogWatch except for the single thing of LogWatch warning about multiple failed login attempts. Anyone is welcome to send me email (bob@verysecurelinux.com) and request a copy of my enhanced Logcheck...

<BobToxen> Everyone should be using Logcheck to monitor their log files (or pay someone like me to do so). My enhanced Logcheck even will generate alphanumeric pager alerts of severe problems 24x7. Of course, even with Logcheck it can be time-consuming...

<BobToxen> Snort also can do popups of problems and I like Snort too...

<BobToxen> Also, for clients with large networks, I install my enhanced Arpwatch that detects unauthorized systems being connected and even misconfigured systems. Quite useful. There are log file monitors for other platforms too that should be used. **

<bdthomas> Ok, last serious question:

<bdthomas> What would you say to a young person who is interested in a career in security?

<bdthomas> Do you recommend any classes, certifications? What is the best way for a newbie to learn Linux security?

<BobToxen> I would suggest reading my book. Ask one's company or school to get it for their library if the cost is an issue. Btw, I only get about $2 per copy so I'm not simply hyping it to get rich. Definitely subscribe to the security mailing list for your Linux Distribution and subscribe to some security mailing lists...

<BobToxen> CERT and the Sans Institute have some good lists. Take some classes in security. I also give free talks every few months and give non-free security courses as well. Becoming aware and thinking "What are the security implications of this thing that the boss wants?" also should be asked frequently. **

<bdthomas> Again, great advice.

<bdthomas> Last question:

<bdthomas> In 2002 when I interviewed you (Here) you said that your only eBay purchase was a Rolls-Royce (Here) Do you still have the car? Has it been reliable? Do you still have the 'LINUX' license plate? Tell us about it.

<BobToxen> BugTraq at securityfocus.com is my favorite security list.

<BobToxen> Yes, I still have the Rolls-Royce and it's still the only eBay purchase...

<BobToxen> Alas, while it has broken down on the road only a few times, not bad for a 25 year old automobile, it requires lots of expensive maintenance that is rather frustrating. It still does have the LINUX front license "plate". I still get stopped and asked about Linux. I also try to send a message about Linux vs. the competition. **

<bdthomas> Awesome.

<bdthomas> Well Bob, I wanted to thank you for participating in today's LinuxSecurity.com chat.

<bdthomas> For those of you interested in his business, see the following:

<bdthomas> Here is the URL to Fly-By-Day Consulting, Inc:

<bdthomas> Linux Security Services, Linux Firewall, Linux Anti Virus Filter, NetworkSecurity

<bdthomas> Atlanta, Georgia, (USA)

<BobToxen> It's been my pleasure. If anyone has further questions, please to send me som email or hit my web site. **

<bdthomas> Buy Bob's Books:

<bdthomas> Here

<bdthomas> Again, thank you. A transcript will be available on Linuxsecurity.com in the near future.

<BobToxen> I'm available for Linux and network security consulting worldwide. My firewalls, VPNs, and virus/spam filters are especially popular. Best regards, Bob **

<bdthomas> ------ END -------