Newer Is Not Always Better

One of the great insights we have at NSS Labs, given the work that we do, is into the trends demonstrated by vendors in terms of performance and security effectiveness across multiple versions of a product.

For some reason, the area we see that is broken most often during a product refresh is anti-evasion measures. Protections that have been in place over several versions can suddenly disappear as a particular section of the code base is updated to include shiny new features. The other major hurdle for many vendors is the move from one hardware platform to another. This often requires major code revisions, if not a complete rewrite, and in this fast-moving industry it is rare that the folks who wrote the original code are still around. The result can be problems in performance and/or security effectiveness that did not exist in previous incarnations of the product.

These deltas and trends across multiple versions allow our analysts to provide actionable advice to NSS subscribers on whether or not to upgrade to a new version of a product, or stay with an older version until problems are fixed (or until it is time to refresh completely and you can investigate other options.)

The response we get from a vendor when we find these issues tells us a lot about how they value their customers over their shareholders, or vice versa. Some are all over the problem and want to work closely with our engineers to discover the root cause of the issues so they can be fixed. Others respond with veiled threats copied to legal counsel, and some serious marketing spin. I would much prefer to see a vendor employ a couple of new developers and fix their problems rather than launch a PR offensive and budget for some legal fees. Either way, we never shy away from publishing the results.

This is why it is important not to simply look at our test reports at purchase time, but also review new reports throughout the life of each security product you have deployed. This can help ensure that no costly mistakes are made in deploying product updates that could have potentially disastrous consequences to your business.

Dogs and children may be for life, but you are not committed to a security vendor in the same way. There is nothing to stop you from doing a forklift upgrade of a product from a vendor that has lost the plot in terms of quality control, and the cost of doing so could be far less than the cost of upgrading to a faulty product just because the vendor wants to EOL the one you have.

Follow me on Twitter (@bwalder) to keep informed as new research is released.