Testing is hard.
For a while there I thought of making that the end of this blog post, but I guess I should elaborate a little. Testing is hard, whether you are a vendor looking to do QA, an independent test lab doing competitive analysis, or an end-user trying to decide which product to buy.
Good test plans are difficult to draw up, and solid methodologies are difficult to create. End-users often use independent reports to create short-lists before doing their own in-house testing or proof-of-concept projects. This is why vendors get so upset when they don't do well in such reports. This is understandable, but what the vendor does next is often a good indicator of character.
The first thing to do, of course, is to verify that problems highlighted in the report are genuine. Vendors should work with the test lab wherever possible and be prepared to do so with an open mind, not get all defensive about the fact their precious product has a flaw. If the test lab can show you time and time again (live or on video) that they owned a target host protected by your product, then you probably have an issue that needs fixing!
Secondly, dedicate some resources to fixing the problem rather than generating marketing FUD to disguise it or deflect attention away from it. Yes, this costs money, whether you do it all in house or engage the test lab to help. Don't expect someone else to fix your product for free!
Third, bask in the glory that comes with fixing a problem quickly and professionally thus leaving your customers exposed for the minimum possible time.
What you SHOULDN'T do is shoot the messenger!
I have seen three examples recently of vendors going on the attack straight away when they don't like what is in an independent report - one in the IPS area, one in Web Application Scanning, and one in AV.
In each case the vendor in question launched public attacks on the various test labs, one of which led Mike Rothman of Securosis to predict the death of product reviews. I think Mike is wrong in this dire prediction, and end-users had better hope that I'm right, because such reviews - when done well - are all that stands between the purchaser and all that vendor hype. That and a Magic Quadrant!
Of course, the vendor is entitled to put forward his point of view. It is not difficult to spot weak methodologies, and these can do more harm than good, and the only recourse a vendor has to to refute the results publicly.
But when you have been caught out, when your product has been shown to have a repeatable flaw, posting falsehoods and ad hominem attacks in an attempt to discredit the report, the methodology, and the engineers who carried out the tests is simply not professional.
The problem is, if the test lab in question DIDN'T foul up the test, you are going to look pretty stupid when they are forced to reveal more and more of the problem in order to dispel your FUD attack. And your customers are going to be upset too, as you dedicate marketing resources to hide an issue better addressed by engineering resources.
If you are a customer of a vendor who engages in these tactics, I would encourage you to make every effort to talk to whoever produced the report which upset them. Try to understand the problem, and make sure that it doesn't affect you. If it DOES affect you, see if they can help you reproduce the tests in your own environment (if it is not too dangerous to do so). At that point you can go back to your vendor with some concrete data, and you will also be in a position to verify any fixes they release for the problem in the future.
I have a series of research notes in the pipeline right now on testing: what you should know, and how to do it properly. It strikes me they are sorely needed!
Wednesday, March 17, 2010
Monday, March 08, 2010
Identity Theft - A True Story To Chill The Heart
It's typical that on the evening before you are about to leave on business for four days you realise your propane tank is empty (there is no mains gas in our village). And you will not be back home until Friday evening by which time it is too late for them to make a delivery before the weekend. And, oh look, the weather forecast has turned to snow by Monday. And so you face a bleak, cold weekend with neither heating nor hot water before they can replenish your gas supply on Monday. Oh joy.
What has this got to do with IAM, you might ask. Nothing at all. But it does give you some idea of my state of mind as I headed north to London to attend the fourth Gartner Identity and Access Management (IAM) Summit - not the happiest, as you can imagine.
But solace was to be found in the warmth of the welcome I received from my colleagues, most of whom I was meeting for the first time in London. And drink. But mainly the welcome...
IAM is a key area for Gartner's clients, of course, and so the agenda was packed with the best and brightest of those Gartner analysts who specialize in Letting The Good Guys In (LTGGI). As a tin-head myself, and part of a separate group in the Gartner Security, Privacy & Risk team tasked with covering technologies for Keeping The Bad Guys Out (KTBGO), I was not actually involved in any of the presentations. Instead, I got to observe my new colleagues in action, brainstorm ideas for research, try to sell them on the fact that if we kept everyone out, Good and Bad, it would make life (and security policy creation) a lot easier, and talk to some of our clients face to face for the first time.
Pretty soon I had forgotten all about propane problems as I immersed myself in IAM-enabled cloud architectures, security monitoring, role & entitlements management, fraud prevention and federated identity management. There were workshops too, many of which were fully booked almost as soon as the summit started, and a constant stream of analysts and clients to and from the one-on-one meeting rooms. Attendee numbers were good, an excellent sign in tough economic times, and everyone I spoke to seemed to be getting a lot out of the event. If you missed it - shame on you. Book early for next year!
Things ended on a light note with a true story of identity theft from writer and comedian Bennett Arron. It all started with a mail-shot from a home shopping catalogue company to an old address, which allowed the unscrupulous person now residing at that address to place an order and open an account with the home shopping company. That credit account allowed him to acquire a mobile phone or two. From there it was not too difficult to open bank accounts and obtain credit cards - all in Bennett Arron's name.
The end result was Arron, who had already given notice on rented accommodation to buy a house, failed to acquire a mortgage, couldn't rent another property, couldn't get a line of credit, burned through savings and ended up penniless and living with parents with his pregnant wife. It took him two years to clear his name, by which time property prices had tripled and he could no longer afford to buy a house anyway! No compensation was forthcoming from any of the companies who allowed a criminal to open accounts in someone else's name, though Arron did get a one-man comedy show out of the material and Channel 4 made a documentary on him. So that's OK then.
One remarkable thing that he demonstrated in the documentary was how trusting people can be when faced with official-looking situations. He donned a suit and tie and set up a stand in a local shopping mall offering people advice on the perils of identity theft. He also offered a free service to protect their most sensitive information provided they would... yes, you guessed it.... give him their most sensitive information.
In just two hours he spoke to twenty people, eighteen of whom happily handed over their name, address, date of birth, credit card numbers, expiry dates and even the 3-character CVV/CVS numbers from the signature strips. Only two people refused. Only one person thought better of it and returned to the stand.
"Hey, this isn't a scam is it?", he asked.
"Errrr.... no"
"Oh, that's OK then. Thought I'd better check though...."
True story!
And just goes to show that no matter how many firewalls and IPS you have installed, social engineering will get you every time.
As part of the documentary Arron attempted to prove how easy it would be to steal someone else's identity. He settled on then Home Secretary Kenneth Clarke, since it would need to be a high profile "theft" to get everyone's attention.
Arron applied for a duplicate birth certificate in Clarke's name, and within 3 days it arrived. Using that, he applied for a duplicate driving license from the UK Drivers & Vehicle Licensing Authority (DVLA), which took just a couple of weeks to arrive. As part of this process, the DVLA requested photographs for the license which had to be authenticated on the reverse with a statement from a trusted, non-family member that this was a true likeness of Kenneth Clarke. This Arron completed himself using a false name. Something of a root trust issue, here, I think....
Naturally, with a birth certificate and driving license Arron could have gone on to open various accounts, building up to bank accounts and credit cards. Scary stuff. One good thing came from this - it is now no longer acceptable to use a birth certificate as the sole means of ID when applying for a UK driving license. Wonder if they have plugged that photo certification loophole too?
As the summit comes to an end and I set off back to my home in France, I reflect on how identity theft would be so much more difficult to accomplish here by virtue of a few simple controls that are universal.
In France, if you want to open any sort of account, from a bank account, through mobile phone all the way down to the humble supermarket loyalty card, you need to provide one piece of photo ID (ironically, a UK driving license is permitted!) and at least one justification of current address. This needs to be something serious, like a bank statement or major utility bill (electricity bill, fixed phone line, but NOT a mobile phone bill).
We often consider these controls to be the bane of our lives here, since they add a layer of complexity to the most simple tasks, but in the light of Bennet Arron's story it makes perfect sense.
Sometimes customer satisfaction is not everything - sometimes you have to put security requirements ahead of signing up that new prospect.
What has this got to do with IAM, you might ask. Nothing at all. But it does give you some idea of my state of mind as I headed north to London to attend the fourth Gartner Identity and Access Management (IAM) Summit - not the happiest, as you can imagine.
But solace was to be found in the warmth of the welcome I received from my colleagues, most of whom I was meeting for the first time in London. And drink. But mainly the welcome...
IAM is a key area for Gartner's clients, of course, and so the agenda was packed with the best and brightest of those Gartner analysts who specialize in Letting The Good Guys In (LTGGI). As a tin-head myself, and part of a separate group in the Gartner Security, Privacy & Risk team tasked with covering technologies for Keeping The Bad Guys Out (KTBGO), I was not actually involved in any of the presentations. Instead, I got to observe my new colleagues in action, brainstorm ideas for research, try to sell them on the fact that if we kept everyone out, Good and Bad, it would make life (and security policy creation) a lot easier, and talk to some of our clients face to face for the first time.
Pretty soon I had forgotten all about propane problems as I immersed myself in IAM-enabled cloud architectures, security monitoring, role & entitlements management, fraud prevention and federated identity management. There were workshops too, many of which were fully booked almost as soon as the summit started, and a constant stream of analysts and clients to and from the one-on-one meeting rooms. Attendee numbers were good, an excellent sign in tough economic times, and everyone I spoke to seemed to be getting a lot out of the event. If you missed it - shame on you. Book early for next year!
Things ended on a light note with a true story of identity theft from writer and comedian Bennett Arron. It all started with a mail-shot from a home shopping catalogue company to an old address, which allowed the unscrupulous person now residing at that address to place an order and open an account with the home shopping company. That credit account allowed him to acquire a mobile phone or two. From there it was not too difficult to open bank accounts and obtain credit cards - all in Bennett Arron's name.
The end result was Arron, who had already given notice on rented accommodation to buy a house, failed to acquire a mortgage, couldn't rent another property, couldn't get a line of credit, burned through savings and ended up penniless and living with parents with his pregnant wife. It took him two years to clear his name, by which time property prices had tripled and he could no longer afford to buy a house anyway! No compensation was forthcoming from any of the companies who allowed a criminal to open accounts in someone else's name, though Arron did get a one-man comedy show out of the material and Channel 4 made a documentary on him. So that's OK then.
One remarkable thing that he demonstrated in the documentary was how trusting people can be when faced with official-looking situations. He donned a suit and tie and set up a stand in a local shopping mall offering people advice on the perils of identity theft. He also offered a free service to protect their most sensitive information provided they would... yes, you guessed it.... give him their most sensitive information.
In just two hours he spoke to twenty people, eighteen of whom happily handed over their name, address, date of birth, credit card numbers, expiry dates and even the 3-character CVV/CVS numbers from the signature strips. Only two people refused. Only one person thought better of it and returned to the stand.
"Hey, this isn't a scam is it?", he asked.
"Errrr.... no"
"Oh, that's OK then. Thought I'd better check though...."
True story!
And just goes to show that no matter how many firewalls and IPS you have installed, social engineering will get you every time.
As part of the documentary Arron attempted to prove how easy it would be to steal someone else's identity. He settled on then Home Secretary Kenneth Clarke, since it would need to be a high profile "theft" to get everyone's attention.
Arron applied for a duplicate birth certificate in Clarke's name, and within 3 days it arrived. Using that, he applied for a duplicate driving license from the UK Drivers & Vehicle Licensing Authority (DVLA), which took just a couple of weeks to arrive. As part of this process, the DVLA requested photographs for the license which had to be authenticated on the reverse with a statement from a trusted, non-family member that this was a true likeness of Kenneth Clarke. This Arron completed himself using a false name. Something of a root trust issue, here, I think....
Naturally, with a birth certificate and driving license Arron could have gone on to open various accounts, building up to bank accounts and credit cards. Scary stuff. One good thing came from this - it is now no longer acceptable to use a birth certificate as the sole means of ID when applying for a UK driving license. Wonder if they have plugged that photo certification loophole too?
As the summit comes to an end and I set off back to my home in France, I reflect on how identity theft would be so much more difficult to accomplish here by virtue of a few simple controls that are universal.
In France, if you want to open any sort of account, from a bank account, through mobile phone all the way down to the humble supermarket loyalty card, you need to provide one piece of photo ID (ironically, a UK driving license is permitted!) and at least one justification of current address. This needs to be something serious, like a bank statement or major utility bill (electricity bill, fixed phone line, but NOT a mobile phone bill).
We often consider these controls to be the bane of our lives here, since they add a layer of complexity to the most simple tasks, but in the light of Bennet Arron's story it makes perfect sense.
Sometimes customer satisfaction is not everything - sometimes you have to put security requirements ahead of signing up that new prospect.
Subscribe to:
Posts (Atom)