(with apologies to that great orator and statesman Roderick Spode)
So, I came across, via the Spire blog, the followup commentary from Noam Eppel to his Security Absurdity: The Complete, Unquestionable,
And Total Failure of Information Security article.
In case you don't remember the original article, the premise was that information security as a discipline has already failed, and the follow-up is more of the same. The argument is predicated on the observation that there are demonstrable failures of security in the world - quite a bit of them as a matter of fact. In other words, his argument is that security (as the applicable discipline) has failed and that we (as practitioners therein) have also failed because of the vast number of security breaches, security issues, and snafu's that occur on a day to day basis. Fraud? We've failed. Phishing? Failed again. Lost luggage? Depends on who lost it, but if it's the TSA - probably our fault.
Now the point that I made the first time around was that it's not productive to define success/failure based on whether or not incidents occur or even by whether or not it's possible for incidents to happen. For example, traffic accidents occur - does that mean that the traffic laws in this country have categorically failed? Could be... or not depending. But folks would never get away with saying this (at least with anyone taking them seriously) until/unless you could prove that the laws were directly related to the number of accidents. In other words, that there was a demonstrable cause and effect between these two things *and* that the particular success criteria used to define "success" (in this case, less accidents) is both relevant and applicable.
In the traffic safety example, the success criteria might be having a low number of accidents. Once you define what it means to be successful, it's possible to measure how people stand up to the yardstick. For example, by comparing the percentage of increased accidents in one area vs. another area with different laws, you can extrapolate as to whether or not the laws in area A are more able to satisfy a given goal vs the laws in area B (for example, maybe there are less accidents.) In this case, the success criteria is whether or not there are incidents; well, if you take a risk management approach, aren't incidents unavoidable? In other words, if I'm only going to spend money protecting resources commensurate with the value of the resource, isn't it implied that there are going to be areas that are less protected than others? For example, if I have ADT in my home, and somebody comes and hits the mailbox with a bat, did ADT fail? *Should* I hire an armed guard to protect the mailbox? Probably not. But if you define success as living in a world where punks can't hit the mailbox, it's a failure.
Maybe we should define success or failure based on something provable and something that works within the context of risk management. Well, I'll stop going down this road, since I covered it all in the original reaction, but I thought it was useful to point out that when you say somebody failed you have to say "what at?" Did the security industry fail at making the world risk free? Unquestionably. Is that the primary goal that we as an industry should be after? Not in this lifetime. How about "reduce the number of incidents to the point that customers are well-served, that money spent by the organization protects resources commensurate to their risk and value, and that we spend enough to ensure that our personal safety is ensured in contexts where it's applicable?" I think that's a pretty good goal... I'm going with that one.
Posted by Ed at January 8, 2007 10:01 AM | TrackBackYeah, I saw that follow-up a while back. I'm still withholding a lot of my opinion until I see what suggestions he has moving forward, as he promised he would post. He made some good points, and some really shaky points, but otherwise so far has just sounded a lot like Steve Gibson...a little pundity, very hyper-active, and looking to just stir the pot and make sure everyone knows it's his hands on the ladle. I wouldn't be surprised to never have a follow-up article from Noam.
You're exactly right, though, about defining success and failure before claiming something a failure. And then evaluating whether that definition is realistic.
We don't yet have world peace, so that means George Bush is a failure (and every other President)! *pounds fist on desk*
Posted by: LonerVamp at January 8, 2007 02:45 PMI recently heard a podcast in which Bruce Schneier used the example of 9/11/2001 to decide whether we succeeded or failed. There were so many things that could have gone either way: warnings, flight delays, etc. What if the plan had been foiled for one of those many reasons? Does that mean our security succeeded? Not necessarily.
The point is, as you alluded to, you can't measure success of security (even if it is defined strictly) on a sample size of one incident. It is such a complex thing that you probably need to use statistics. Then you get into arguments about whose agenda the chosen statistics are serving. Sound a bit like Global Warming? ... Oh, yeah. That's our fault, too. (Canadians, that is. You know the "cold Arctic air mass coming down from Canada"?)
Posted by: Scott Wright at February 8, 2007 10:55 AM