Testing Times In Security

I am speaking to more and more enterprise clients who are doing their own in-house testing of security devices. Some of them invest in large, dedicated test networks and knowledgeable personnel, others invest in a single rack of virtualization and load generation equipment. But for all of them, the aim is the same - reduce risk of compromise by throughly testing equipment against enterprise-specific criteria before purchase.

Security vendors' marketing claims are often exaggerated, and frequently do not reflect real-world or enterprise-specific conditions. Performance of complex network security devices is difficult to determine accurately, yet failure to do so can result in significant negative impact on the network should the wrong device be selected or a chosen device configured incorrectly.

Testing is not necessarily about proving that the most-capable, most-expensive product is the best choice. A well-designed testing plan may actually show that a lower level of performance is acceptable at certain points on the network, and this can reduce purchase and deployment costs. IT organizations that do not perform relevant tests in- house may introduce serious security and performance issues to their networks by purchasing underspecified devices, or may overspend significantly on higher levels of performance and coverage that are not required.

Security effectiveness of complex security devices is often the most-difficult area to evaluate, because it requires expertise with attack traffic, and even live exploits. Evasion testing in particular seems to be a challenge for even the best-equipped enterprise test labs (hardly surprising, since it also appears to be something of a challenge for many of the security vendors out there!) For those with the requisite expertise in-house, however, a basic security effectiveness test bed can be created at a relatively low cost using virtualization technology and commonly available test tools. Virtual machines can be used to create an environment that is safe and repeatable, allowing security-conscious organizations to verify the often inflated vendor marketing claims.

Although it requires little in the way of specialized expertise and test equipment, testing the user interface (UI) and device management capabilities is often overlooked when evaluating complex network security products.

This is a mistake, however. A management system that does not meet organizational requirements reduces the effectiveness of a security solution. If a task is too difficult to perform, then it will be executed poorly or inconsistently, if at all. Operational costs can also be reduced drastically via well-designed centralized management systems.

Those who take testing seriously also implement continuous testing programs, making them an integral part of the ongoing security maintenance regime. I have seen numerous instances in the past of a single poorly written signature crippling the performance of an IPS. Firmware updates can also break previously solid inspection processes — anti-evasion techniques appear to be particularly prone to disruption between firmware updates.

Once initial deployment of your security device is complete, perform a full benchmark test to establish a baseline for your existing deployments. Every time a new firmware upgrade, signature pack update or change in security policy is applied — however minor it may seem — the device should be retested and the results compared against the baseline. In-place, ongoing penetration tests on the live network can also help to identify changes in security effectiveness following updates. This process of continuous monitoring makes it possible to monitor, identify and correct adverse impacts on performance or security effectiveness.

We currently have a number of ANalysis Briefs in the pipeline covering performance testing, security effectiveness testing and managing security devices. Together these will provide you with plenty of background material gleaned from almost 20 years in the security testing industry, along with some actionable advice to help you avoid costly mistakes when selecting and implementing complex network security devices.

Don't forget to follow me on Twitter (@bwalder) to be kept informed of new research.