Who Pays For Testing You Can Trust?

This is a question often overlooked both by those who scream "bias" and those who cry "but I want all my information for free!"

The point is, should you stop and think about it for more than a minute, there is no such thing as a free lunch - or a free independent test report. Someone, somewhere, has to pay for it. And at the end of the day, the test lab has to make a living, and there are only three ways it can do that:

1. Free testing, free reports, money comes from advertising

2. Money comes from participating vendors - reports are made available for free

3. Testing is free to vendors, end-users have to pay for reports

That's it. Those are your choices. And in all honesty, there is no difference between options 1 and 2, except that advertising revenue is hard to come by and the tests are never likely to be as thorough as you would like. Option 1 is the magazine model, and we can ignore it when discussing independent test labs.

So the proper labs are left with two choices - vendor pays or end-user pays.

First question is, does the fact that the vendor pays for the test devalue that test in any way? The answer is, "it depends on the integrity of the lab". If the lab prepares a solid, vendor-agnostic test methodology and sticks to it and reports all results, warts and all, for all vendors in the same way, then the model works just fine. Where the vendor (or in some cases, a consortium of vendors, even when watered down with tame test labs) gets to define the test methodology, or veto test methodologies it does not like, then there is something rotten in the state of Denmark. Avoid reports that come out of such a process.

You can usually sniff out the best methodologies - look for the ones that are open, thorough, published, clearly vendor-agnostic and which result in tests which are repeated time after time in the same way. Avoid "methodologies" which are aiming for the lowest common denominator, or which are "one-offs", clearly specified by a single vendor to show their product in the best light. How can you spot those? Simple - they are indeed one-offs, and you will never see that test methodology used to test another product. Labs should have a different test methodology for each product category - watch out for the ones which have a different methodology for each vendor!

I speak from experience here, having spent almost 20 years in the testing and certification business before joining Gartner. Now personally, I never used to accept single-vendor sponsored reports. Not because I wasn't confident I could still do the same rigorous, independent test, but because of the perception. If the vendor concerned doesn't like the report, he gets to squash it - that's his right as the commissioning entity. But if he does well in the test, then he will be more than happy to publish the results. Unfortunately, no matter how scrupulous the tester and the testing process, anyone who doesn't like what the report has to say (other vendors, or end users who purchased competing products and don't like that their choice was not validated publicly) will cry - usually loudly and publicly - "well of course they were bound to win - they paid for it!" Very unfair on all concerned, but almost inevitable.

Group tests usually work better, since even when the vendors are paying, it is obvious that they are a) all paying the same, and b) they are all being tested and reported under the same methodology. Unfortunately, the vendors still usually get the option to squash reports which can have the unwanted side-effect of a group test of 12 vendors resulting in a finished report containing only 2! In addition, vendors can hide behind budgetary issues as an excuse for non-participation.

This brings us to option number 3. This is a huge gamble for the test lab, which can spend months testing products only to find that sales do not cover costs. But the advantages are clear. They can dictate who is tested and can include vendors who would prefer not to participate because of technical issues. This approach is fine as long as the vendors are given the option to provide technical support and ensure their product is correctly configured.

As with the paid group test, everyone is treated equally and the results are reported warts and all. This time, the vendors don't get the option to pull out of the test if they do badly, of course, and this can result in some nasty repercussions for the lab. Vendors who do badly will go on a massive PR damage limitation offensive which will include some very public denouncements of the process and findings. Sometimes these attacks are not so public, aimed at existing customers via private communications, making it almost impossible for the lab to defend itself against unfounded allegations.

The end result, however, is a report which is much more valuable to the end user and potential purchaser of the products under test. The down side, of course, is that now it has to be paid for! C'est la vie. You can't have your cake and eat it too!

The vendors, too, must learn that they cannot have it both ways. If they do not want to pay for testing up front, then when the lab finds problems with their product what can they expect for free? Certainly the lab should tell them what they found and why the product did poorly. But how much information are they obligated to provide? Surely that is the extent of it? Should they be expected to act as an unpaid QA facility for vendors? Or should they - should we all - expect that these products do what the vendors claim, and if they don't they need to be fixed at the vendor's expense?

The vendor is always at liberty to go away and invest in research and technical staff to reproduce the bugs or problems found. Or it can choose to pay for consultancy to expedite that process. I keep seeing vendors complaining in public forums about how they did poorly in tests and the test lab won't provide them with all of their test material to reproduce the tests.

Well why should they? Shouldn't that be considered their intellectual property? Should they not be recompensed for helping vendors fix these glaring errors? How do you as end users feel about vendors which will not invest in their own QA process but expect external entities to do it for free?

Who amongst us here is willing to work for free? It is not a widely accepted concept - don't apply it to others unless you are prepared to do it yourself.