The following article is excerpted from the Computer Security Journal, Vol. XII, No. 1, Spring 1996. All rights reserved.

How to Perform Effective Firewall Testing

By E. Eugene Schultz, Ph.D.

Firewalls are barriers at the gate to a network that filter the transmission of certain types of traffic according to security criteria.1 Few measures have contributed as much to network security, particularly Internet security, as firewalls. Despite many limitations,2 firewalls can deter a wide range of network attacks.

The technology of firewalls has advanced considerably. Whereas the first generation of firewalls was relatively simplistic, most vendors now make at least reasonably good firewall products that adequately accomplish the purposes for which they were designed. We have in the process overlooked one very important question, however. How can one know whether or not a firewall is effective? Answering this question necessitates firewall testing.

The practice of firewall testing is not nearly as advanced as designing and implementing firewalls. Vendors and consulting services seldom bill firewall testing services as a principal offering. Worse yet, a reasonable proportion of those who perform firewall testing know something about the technology of firewall testing, but fewer genuinely understand the process of firewall testing. Furthermore, firewall testing is, in many respects, unlike normal security testing, such as systems penetration testing. The use of technology alone does not result in effective firewall testing. The most brilliant technical person in the world can use the most advanced technology available anywhere, yet fail to provide an adequate firewall test. This paper discusses what constitutes an effective firewall test by covering the nature of firewall testing, the goals of testing, proper testing procedures, and who should perform the testing.

The nature of firewall testing

A firewall test is first and foremost a type of penetration test. A penetration test uses techniques designed to defeat and bypass security mechanisms to determine the effectiveness of such mechanisms. Because penetration testing, whether aimed at systems, networks, or firewalls, utilizes methods that intruders would use, it is a two-edged sword. On the one hand, it provides the ultimate test, a "rubber meets the road" type of test, in many respects. A firewall that can withstand a real attack justifies confidence in the level of security control it provides. On the other hand, few matters provoke management's ire and disturb the political equilibrium as much as penetration testing efforts that breach security control mechanisms, revealing inadequacies in an organization's security infrastructure. Additionally, penetration testing can go awry. A penetration testing team, for example, virtually brought down a computing operations center at a military site because of careless testing procedures, resulting in substantial financial loss because of computer down time, in addition to personnel time needed for recovery efforts.

The point is that effective firewall testing is much more than simply unleashing a series of attacks, then determining whether or not they defeated the firewall's security mechanisms. Too much is at stake, and the value of firewall testing can be greatly diminished if it is not conducted properly. To be done properly, firewall testing should involve at least as much planning as the testing activity itself. Planning is the most basic step in effective firewall testing.

Basic issues in firewall testing

The first step in firewall testing, then, is to establish ground rules for testing in much the same way that one should develop a firewall policy before developing a firewall.3 The most fundamental issues to be considered are:

Rationale. Firewall testing is not a benign activity; if not done properly, it can backfire and cause disruption and political backlash within an organization. Does a genuine need for firewall testing exist or might a thorough analysis of the firewall's security features and vulnerabilities be sufficient? The primary motivation for testing a firewall should generally be the need to determine, through empirical methods, the firewall's ability to prevent the kind of attacks that intruders are likely to perpetrate.

Management cognizance and approval. Fire-wall testing that occurs independently of management cognizance and approval is likely to end in catastrophe. The testing will almost certainly be viewed as a renegade effort if management is not involved. The recent case4 in which a contractor was convicted on multiple felony counts for obtaining password files from a corporation's UNIX machines, then running CRACK (a password cracking tool) without management approval aptly illustrates the importance of this issue. Without written management approval, plans to test a firewall should proceed no further.

Disruption. Firewall testing can possibly disrupt ongoing network operations. Attack scripts used in firewall testing can overload a network with service requests and tie up ports on host machines. Constraints concerning tolerable levels of disruption should be set before any testing begins.

Pre-notification. Notifying network administrators and others in advance of an impending penetration test and possibly of the time of the planned test reduces the likelihood that they will detect the activity and, thinking that an attack on the firewall has occurred, devote a considerable amount of wasted time and effort to investigate. On the other hand, pre-notifying network administrators about an upcoming firewall test can enable them to change settings in firewall configuration files and kill dangerous services the firewall runs so that the firewall can withstand attacks, then change settings and services to their original condition immediately after testing is complete. In the latter case, the firewall test's value has been diminished; in-creasing the security defenses of the firewall only when firewall testing occurs, then relaxing these defenses afterwards, is an unrealistic test of the firewall's operational security condition. No easy answer to the dilemma regarding pre-notification exists.

Supervision. Should the person who performs firewall testing be supervised? Some organization's security policies offer an easy answer to this dilemma by requiring the presence of an employee whenever a consultant obtains access to that organization's computing resources. An advantage of supervising testing activity is learning exactly what is being done and how the firewall has withstood attacks launched against it. Another advantage is ability to reasonably ensure that the person performing the testing does not do anything unethical or reckless. On the other hand, a supervisor can slow down testing considerably and may even get in the way, causing the person doing the testing to perform steps incorrectly and more slowly. The resolution depends primarily upon how critical the firewall and network tested are; the more critical, the more the need for supervision exists.

Use of results. Management and technical personnel should determine how the results of the firewall testing activity will be used before any testing actually begins. Ideally, the results will be used to provide an objective indication of the security condition of the firewall that is tested to improve the security if necessary. Firewall testing can also be used to aid in the decision whether to continue using a firewall product or replace it with something that may be better or to decide whether to increase host-based security mechanisms.

Safeguarding results. Making provisions to safeguard results long before firewall testing commences is another necessary step. The results of a firewall test in which firewall defenses are breached can serve as a roadmap to a devastating attack on an organization's network. If not adequately protected, results could end up in the hands of a disgruntled internal employee or could even be posted to a net group on the Internet. Rival political organizations could also obtain the results and use them as leverage against the department or function that simply wanted a reality check on the condition of their firewall and nothing more. Designating the employees and consultants allowed to possess copies of the firewall results, putting restrictions of copying and disseminating these results, ensuring that a safe storage capability exists, and similar measures can preclude undesirable complications.

Goals of testing

Assuming that the decision to proceed with firewall testing has been made and the ground rules of firewall testing have been established, the next step is to determine specific goals of the firewall testing effort. The goal at the highest level is to observe whether security defenses are adequate when a firewall is actually attacked. More properly, the firewall should not only be able to withstand a direct attack, but the firewall should be able to prevent successful attacks against hosts within the security perimeter created by the firewall. In firewall testing, however, high-level goals are insufficient because they merely reflect the overall purpose implied in the name "firewall testing." In contrast, developing specific goals is important because these goals actually guide the testing process, yielding results that are likely to be of maximum value to the organization. Specific goals determine the type of firewall testing that is appropriate and the types of procedures that should be used, serving as functional requirements for the firewall testing process. The following questions can be used to formulate specific goals:

Is the firewall a correct implementation of the firewall security policy?

The policy should define the services that are and are not allowed, and, if the firewall also controls remote connections to applications and services, how these connections are controlled. The policy may, for example, state that all incoming mail traffic will be allowed, but that the firewall will proxy mail services to a mail server within the internal network. This policy requires that the firewall receive all incoming mail, but forward any mail it receives to the mail server. The policy may also state that all incoming ftp packets are not allowed. If during the course of testing the person conducting the test discovers that the firewall allows modification of mail exchange records or accepts incoming ftp packets, something is wrong-the policy is not in this case correctly implemented.

How well does the firewall resist particular types of attacks, particularly new types of attacks?

CERT5 and major UNIX vendors frequently announce new security vulnerabilities and applicable patches. Many of these new vulnerabilities surface after the firewall is implemen-ted. Attempting to breach firewall de-fenses by exploiting each new vulnerability yields answers concerning the firewallis ability to resist new attack patterns.

Does leakage occurs in the security perimeter created by a firewall?

Leakage results when access routes into a network other than through a protected gate exist, thus allowing someone to bypass firewall defenses to reach this network.6 Leakage occurs more often than most people suspect. Discovering not only that it exists but also the ways in which it exists is an important finding of a firewall test.

How adequate is the logging capability?

This capability is especially important because of the nature of many attacks on firewalls. Attackers often patiently gather information about the firewall and connection routes to the firewall from other hosts, then launch attacks at well-spaced intervals to make detection less likely. Firewalls that log all incoming (and outgoing) requests, including information requests, are more conducive to quickly and efficiently discovering attacks so that countermeasures can be taken.

Does the firewall have the ability to send an alarm?

Ideally, firewall testing activity will trigger alarms shortly after penetration testing commences. The alarms can be in the form of a message sent to network administrators, a pager beep, or some other event that gets some cognizant person's attention. A test performed on a corporation's firewall about ten months ago not only caused messages to be sent to the network administrator and corporate security manager, but also a message to root on the machine from which the test was launched. The later message stated that suspicious network activity had originated from that machine and provided the name and telephone number of the network administrator.

How well does the firewall hide information and addresses from the internal network it protects?

Answering this question is the last of the issues that govern the specific goals that are chosen. An effective firewall will, for example, strip the address of the host from which an e-mail message is sent and replace this address with the firewall's. A firewall that does not strip the originating address does not hide names of internal hosts; attackers can thus learn these names and launch attacks against specific hosts. Achieving this goal does not require penetration testing per se,7 but determining the type of information available outside of an organization's security perimeter is an important supplementary finding to the results of penetration testing. Information concerning which mach-ines within the security perimeter are server machines, for example, generally should not be externally available for security reasons.

Testing procedures

Having detailed, written procedures that everyone can easily follow is essential in virtually every area of information systems security, and firewall testing is no exception. Among the values of effective procedures are helping ensure that each step of firewall testing is performed correctly, preventing omissions, and ensuring replicability of procedures and findings.

The first step in developing suitable firewall testing procedures is creating a test plan.

This plan should specify the scope of the test, the firewall and hosts to be targeted,8 the goals, and the steps to be taken to achieve each goal. If the purpose of the firewall test is to find leakage, for example, the plan should specify a subset (or possibly the entire set) of systems within the internal network to be tested and the approach to be taken, i.e., a direct attack or another tack.

Once the test plan is complete, the person performing the test should develop specific procedures to be used. These should include specific steps such as using the traceroute command to find alternate routes to hosts within the internal network, "war dialers"9 to local modem dial-in numbers within this network, and similar procedures. Determine the parts of the firewall test that should be performed interactively and the parts that are best performed using automated attack scripts.10

Effective firewall testing generally requires both interactive and automated testing. Use interactive testing whenever there is a possibility that events may get out of control quickly or when the need to observe the results of each action very carefully exists. Some attacks (e.g., attacks that capitalize upon timing conditions) work best when they are programmed into attack scripts. Executing attack scripts is also less time-consuming. Be sure, furthermore, to test every attack script in a non-production environment to ensure that it does not cause unintentional damage or disruption.

How far should firewall testing proceed? Should one get as deep into a firewall and systems it protects as possible, showing how critical configuration files can be accessed and possibly changed? Should one produce copies of critical system files such as the UNIX /etc/passwd file as proof of penetration? The answer depends to a large degree upon the ground rules and goals of the effort. If the goal is simply to determine whether or not vulnerabilities exist, going this far does not make sense.11 Testing to the point of conclusively demonstrating that one or more vulnerabilities exist without fully exploiting each vulnerability is, in fact, the safest approach. On the other hand, management sometimes needs to be shocked into reality through firsthand demonstrations of what actually can be accessed and/or modified in the firewall and systems it protects. Regardless of the position one takes with respect to the depth of testing issue, a good principle to practice is to never do anything that is likely to significantly disrupt network operations and critical processing activities on systems.

Part of the process of using sound procedures is carefully documenting a firewall test. A person who performs a firewall test should record every action taken, the results, the name and IP address of both the source and target host, and other critical data and observations. Sometimes obtaining a printout is the most logical method of producing documentation, but don't forget to carefully add missing details. Carefully recording observations produces a record on which to base conclusions and recommendations and preserves details that may initially seem insignificant, but later turn out to be critical. Finding that the target firewall runs a particular service such as rusersd, for example, may not initially seem important. A security vulnerability in this service may surface later, however, rendering this finding much more critical. Good documentation can also help the person performing the testing readily discover omitted or partially completed steps.

Communicating results of a firewall test is the culmination of firewall testing activity. The best deliverable is a report that documents this activity and presents recommendations for improving firewall security control measures. Writing a report may seem anti-climactic after all the testing activity is complete, but a poorly written report greatly diminishes the value of a well-planned and executed firewall test. This report should meet the following criteria:

Objectivity. The report should objectively describe the methodology and findings, and should avoid speculation such as, "Additional mail ex-ploitation scripts not used in the current test may have allowed re-mote root access to the firewall…" Do not confuse speculation with findings.

Thoroughness. Including details about testing that you have recorded is useful to technical personnel who will refer to the report and will make the report worth archiving.

Clarity. The content should be highly comprehensible to all who read it. Define technical jargon either using footnotes or by providing a glossary at the back of the report to enable those who are not technically proficient to learn and benefit from the report.

Well-organized. A well-organized report is more understandable. Additionally, readers may be more interested in some sections of the report than others, and may need to locate certain content quickly. Organizing the report into clearly labeled sections such as background, methodology, findings, conclusions, and recommendations makes the report more useful to those who read it.

Solution-oriented. Presenting a solution or solutions with every security weakness documented in the report points the organization that administers the firewall to a positive outcome. Presenting advantages/disadvantages and possibly even prioritizing each solution (starting with the fixes for the most glaring security holes that can lead to compromise of the entire network) can be extremely advantageous.

The last criterion merits further discussion. Some firewalls provide excellent security control such that firewall testing efforts fail in obtaining access to the firewall and systems within the security perimeter the firewall creates. The report should in this case give credit where credit is due. If the firewall can be penetrated or by-passed, the report should motivate the organization that operates the firewall to fix vulnerabilities and state how to do this. The ultimate aim of a penetration test is to improve firewall security where it should be improved. A well-written report does much to help accomplish this goal.

Who should perform the testing?

The next consideration in firewall testing is who will actually perform the testing. The first inclination many people have is to choose the vendor who sold and installed the firewall in the first place. This choice, unfortunately, can often turn out to be one of the worst choices. Firewall vendors often perform some tests on the firewalls they have recently installed as part of the installation process, but these tests are often not very complete. Some vendors, for example, run a port scanning tool to determine whether traffic destined for certain ports is blocked. This simple test can provide some useful information, but this level of testing produces no feedback concerning the way connections allowed are managed for the sake of security. Many eminently qualified firewall developers are not good firewall testers. Vendors of firewall products, furthermore, are frequently not aware of vulnerabilities in their own products, and even if they are, they may not be motivated to announce that they found a glaring security hole in their product. In addition, developers form biases about the way firewalls should and should not work on the basis of their own creations, so to speak. These biases can result in an unduly limited scope of testing in which avenues that a bona fide intruder might pursue are not investigated during firewall testing. Exceptions to every rule exist; some firewall developers are good firewall testers. Choosing someone who installed the firewall to be tested is, nevertheless, not necessarily a good idea. Another tempting possibility is to hire a bona fide network attacker, ex-network attacker, or a close associate of a network attacker. After all, who knows best about defeating firewalls and breaking into systems? This solution is the worst of all, however, because it puts an organization at the mercy of someone with poor or questionable personal integrity. Authorization to perform firewall testing allows someone to attack a firewall and the hosts behind the firewall, something that constitutes a major bond of trust. A known network attacker or close associate simply does not deserve this level of trust, regardless of the degree to which this person now claims to be reformed. Numerous case studies of corporations that have suffered a significant financial loss and disruption to computing resources because of hiring a network attacker reinforce this notion. Another temptation is to hire a known "cybercop." This approach is once again no panacea in that it provides no guarantee of firewall testing expertise per se.

Who is the right person to perform firewall testing? Consider the following criteria:

Integrity. First and foremost, the person (in addition to the organization this person represents) who performs firewall testing must be honest and trustworthy (as discussed previously in this section).

Independence. Hiring a person who maintains an independent perspective is also extremely important. Whomever is hired to perform firewall testing should be motivated solely to show how adequate the firewall is for the purposes it is intended to serve. In this respect not only firewall vendors but also internal employees (who invariably play some part in organizational politics) may not be the best candidates to perform firewall testing.

Experience. Consider only candidates who have considerable experience in this activity and know how to plan and execute successful firewall tests. "Rookies" are not likely to know enough to find security weaknesses in firewalls, even if many weaknesses exist.

Technical capability. Understanding how firewalls and systems behind them are attacked is prerequisite to firewall testing. The person who performs a firewall test must use the range of attack methods that occur on the Internet and elsewhere.

Planning. A well-planned firewall test is most effective. Ask candidates for firewall testing what they will do to prepare for testing activity and ask to see any written procedures. Select an individual who demonstrates a strong orientation towards careful and competent planning.

Communication skills. As mentioned previously, performing firewall tests is only one part of the total process of firewall testing. Communicating methods used, findings, and recommendations is an often overlooked part of firewall testing. Unless the results of firewall testing cause positive changes in an organization's security infrastructure, the value of this testing is minimal. Effective communication is often the difference in motivating management and network administrators to make needed changes based on results of a firewall test. Observe how carefully each candidate for penetration testing listens and reacts to what is said. Ability to express oneself accurately yet tactfully is another highly desirable skill.

Cost. Some vendors charge a package price for firewall testing; others charge by the hour. Rates vary considerably. Be sure to understand how much firewall testing will cost before hiring anyone to do it. Cost should not be the only factor in deciding whom to choose, but some people who are qualified to perform firewall testing can plan, execute and write-up a firewall test much more efficiently and with better quality than others.


This paper covers a wide range of considerations concerning firewall testing. The main point is that firewall testing conducted for the sake of firewall testing accomplishes little. Firewall testing needs to be well-planned and methodical, conducted in accordance with carefully considered goals. Adopting a perspective similar to the one a good software engineer takes by defining requirements, establishing sound procedures, performing all steps of testing methodically, and documenting all actions and outcomes is the key to effective firewall testing.

A discussion of firewall testing would not be complete without discussion of a few final issues. First, attempts to defeat a firewall may be unsuccessful, but these results do not at all imply that the target firewall was impenetrable. Passing a firewall test so to speak may, in fact, lead to overconfidence in a firewall's ability to withstand attacks. The person performing the test may simply not have launched a sufficiently rigorous set of attacks. Alternately, perhaps network administrators temporarily tightened the firewall's security for the testing. If the firewall demonstrates no security weaknesses whatsoever, take the time to investigate why.

Second, firewall testing is useful, but not as useful as firewall testing combined with a security analysis of the firewall. A firewall test may be unsuccessful in discovering the particular services a firewall runs, but subsequent inspection of the firewall's configuration files can reveal that the firewall runs unnecessary and dangerous services, has numerous user accounts, and so forth. Furthermore, conducting some kids of tests may be undesirable. Testing for susceptibility to IP spoofing attacks12,13 provides a good example; targeting hosts within a network protected by a firewall is usually not practical because IP spoofing disrupts ongoing network operations. Inspecting the firewall or external router to determine whether incoming packets bearing IP addresses of hosts within an internal network are rejected is thus better than performing an actual IP spoofing test.

Third, remember that the threat of social engineering also looms continuously.14 Even the strongest of firewalls is subject to compromise through social engineering.15 Attempting to social engineer network administrators and others to obtain information needed to defeat a firewall may in some cases be a logical supplement to firewall testing. Remember, however, that legitimate social engineering attacks initiated by consultants and/or employees can cause considerable resentment and suspicion.

Finally, firewall testing is much more than a one-time activity. Attack patterns change over time; firewall defenses are often relaxed for the sake of network performance enhancement. Conducting firewall testing at regular intervals three to six months apart has thus become essential in determining the security capability of these critical tools and making appropriate changes.

The assistance of Terry Bernstein in critically reviewing this paper is gratefully acknowledged.


1. Cheswick, W.R. and Bellovin, S.M. (1994) Firewalls and Internet Security: Repelling the Wily Hacker. Reading, Mass.: Addison-Wesley.

2. Schultz, E.E. (1995) "A New Perspective on Firewalls," Proceedings of The Twelfth World Conference on Computer Security, Audit and Control, pp. 22 -26.

3. Power, R. (1995) "CSI Special Report on Firewalls: How Not to Build a Firewall," Computer Security Journal, Vol. 9, Issue 1, pp. 1 - 10.

4. Oregon vs. Schwartz (1995) Criminal case tried in Washington County, Oregon.

5. The Computer Emergency Response Team at Carnegie-Mellon University.

6. Cheswick & Bellovin, 1994. Chapman, D. B., and Zwicky, E. (1995) Building Internet Firewalls, O'Reilly and Associates (Sebastopol, CA.).

7. Effective firewall penetration testing involves more than simply attacking the firewall and the systems it protects. It includes systematically determining how the firewall protects everything it is supposed to protect.

8. Obtaining written authorization to test every host included in the test plan is a good idea. Failure to obtain pre-approval can cause the same kinds of problems as failing to obtain management authorization to perform testing.

9. "War dialers" are programs that dial one telephone number after another to find numbers of modems.

10. Ensure that any attack scripts used are well-guarded, so that they do not fall into the wrong hands.

11. Remember, however, that some widely available scripts such as SATAN (Security Analysis Tool for Auditing Networks) can detect only a very limited set of vulnerabilities are thus not very suitable for most firewall tests.

12. In an IP spoofing attack an intruder establishes a connection to a server from a host that masquerades as a legitimate client.

13. Thomsen, D. (1995) "IP Spoofing and Session Hijacking." Network Security, March, 1995, pp. 6 - 11

14. Social engineering is using deception to obtain information to be used in attacking systems and networks.

15. Winkler, I.S. & Dealy, B. (1995) "Information Security Technology? … Don't Rely on It: A Case Study in Social Engineering," Proceedings of USENIX Security Symposium.

Dr. Eugene Schultz SRI Consulting is the Program Manager for SRI International's International Information Integrity Institute (I-4). An expert in UNIX, network security, and security management, he has testified about intrusions into U.S. military computers during Operation Desert Storm before the U.S. Senate. He has also helped various agencies and corporations create information security policies and technical security practices, and regularly performs penetration and firewall tests for clients.

Dr. Schultz has co-authored the IIA/EDPAA book, Unix-Its Use, Control, and Audit, the soon to be released John Wiley book, Internet Security for Business, and has published over 60 journal articles. Before joining SRI he was at Lawrence Livermore National Laboratory, where he founded and managed the Depart-ment of Energy's Computer Incidence Advisory Capability (CIAC), a team dedicated to responding to incidents at all Department of Energy sites. He also held positions at the Jet Propulsion Laboratory (where he received a NASA Technical Innovation Award in 1986), Arca Systems, and the University of North Carolina, and recently re-ceived the Best Paper Award at the 1995 National Information Systems Security Conference.


Copyright © 1998, Computer Security Institute, 600 Harrison Street, San Francisco, CA 94107. Telephone: (415) 905-2626 Fax: (415) 905-2218. Please send us your feedback.