Folks respond to yesterday’s snark (or “Ed makes enemies, part 582″)

Posted by Ed in Analysis on Jun 17, 2010

OK, so I thought it was worth mentioning that AMTSO blog responded (in part) to my snarky post from yesterday about the new guidelines for anti-malware testing – they talk about other things as well, but to us in part. Anyway, their response was not favorable as you can probably guess. So I thought it would be useful to clarify my position about the issues.

#1 – Vendor Involvement

First, I have an issue with vendor involvement in setting standards for testing. So David (the author of the blog article in question) charges that I mis-cited the degree of vendor involvement in the forum. I cited the founding press release for their membership data rather than current membership (I used that list for brevity), but I think the point is still valid: the current membership breakdown is: 70% vendors, 5% individual members, 2% press, and 21% testing organizations. In other words, the makeup of the organization is greater than 0% vendors. This, in my opinion, is an issue.

So what’s my beef here anyway? Why do I care if vendors participate?

It comes down to the purpose of the organization. If the purpose of the organization is to define testing standards for unbiased, repeatable testing of products (see their charter), then I think any number greater than 0% is inappropriate. Why? Conflict of interest. In other words, I think there are good reasons to not have the manufacturers of the products fund the development of testing or safety standards for the products they develop.

It’s not that I think vendors are evil; I’m not sure that vendors would necessarily – or even deliberately – skew the process in their favor (that is a possibility – though not entirely likely), but more to the point, they have an inherent bias. For example, we wouldn’t expect car manufacturers to define safety or emission standard for vehicles. Sure, as an industry they can self-regulate over and above other regulation – but if the interest is public safety (as I think is the case with AV), the standards need to be unbiased. And for the standard to be relevant, appearance of bias is just as important as the fact. It’s the same reason Consumer Reports doesn’t accept advertising.

#2 – Malware Creation

So, my second point is that I think creation of new malware for the purposes of testing is not unethical. David charges that we didn’t read the full analysis of the pros and cons before making our remarks. I’ve read it, I just don’t think it’s germane to the discussion. My point doesn’t rest on whether testing is improved by having new malware or not (full disclosure: I happen to think it is). What concerns me is not arguments for or against, it’s the impact of the statement on non-AV vendor testing.

The AMTSO didn’t just say that they were against creation of malware for testing (well they said that too) – but instead what they said was, ” …unanimous disapproval by AMTSO of the idea of the creation of new viruses or other malware”. Their statement is not limited in scope to just testing. It could have been; they could have chosen to say, “unanimous disapproval by AMTSO of the idea of the creation of new viruses or other malware [for the purposes of testing]…” but they didn’t. By choosing not to limit the scope, they are saying (as I read it) that it is unethical to create malware for any purpose. Period. For academia – for the press – for research purposes – for any reason.

For example, if an educational institution wants to encourage students to think out of the box about malware – and maybe create some as a thought exercise – that’s unethical. See, I think the issue is about what the AMTSO is – or where they’re going. You see, I think they are shaping up to be the AV industry’s industry-recognized standards body. I’m not sure if that’s what they intended at the outset, but that’s where this is going.

See that terminology there, “industry-recognized standards body” – since we don’t have one and are in dire need, they are de facto it. I’m fine with it going that way, but it carries with it a heavy responsibility for them. Chief among those is the responsibility to not quash innovation, to foster academic exploration, and to not hamper research. As an accepted standards body, them putting out a statement about the ethics of doing XYZ (doesn’t really matter what – in this case creating malware) or saying that doing so is contrary to the interests of public safety carries weight. As a consequence, Sonoma University is going to have a harder time justifying their malware class, Consumer Reports is going to have a harder time pursuing their testing methodology, etc.

I’m fine with them limiting the scope of “best practice” testing. I’m even fine with them defining standards for how to contain malware in a lab context. I’m not OK with making it harder for others to do research or innovate.

David Harley

1) I think I made it very clear in that post that I was not speaking for AMTSO, but expressing a personal opinion, so your insistence on referring to it as AMTSO’s response is misplaced.

2) You implied that all the organizations in that list were AV vendors. That may be not have been intentionally misleading, but it was misleading.

3) I don’t agree with your point that vendors are not entitled to a voice in defining good practice, but I don’t have a problem with your holding that opinion. If this was about defining testing industry standards, I’d even agree that the standards shouldn’t be defined by the AV industry, but I wouldn’t agree that the AV industry shouldn’t have any input into those standards. And that’s not the way other testing industries work, in general.

And yes, I know the organization has the word standards in its name: personally I would prefer it if it didn’t, because it’s not in a position to set standards in a BSI/ISO sense. I think that would have to be role of another organization, so to that extent I agree with you. And while I don’t agree that AMTSO should be a vendor free zone, I agree that it’s unfortunate that there aren’t more testers and publishers, and I suspect that most of us would like to see a better balance. That doesn’t mean, by the way, that there couldn’t be or shouldn’t be an organization that is a vendor-free zone. But right now, there isn’t. Unfortunately: if there was, there’d be less need for AMTSO and I might get some of my life back.

4) So you did read the malware creation document, but chose not to refer to it. That’s your privilege, but it’s misleading, and I hope your readers will read it and make up their own minds. Yes, the members of AMTSO (including a number of professional testers) chose not to restrict that statement in Principle 1 to testing, but that’s because most of us think that way. However, I (not speaking for AMTSO, remember) think that for most people, the argument that a competent tester is unlikely to _need_ to create malware is far more compelling than the ethical argument. I don’t expect you to agree with it, but I don’t think you should ignore it.

5) So why do you have a problem with Consumer Reports justifying what they do? The security industry expects to be accountable to its customerbase for what it does. Why shouldn’t testers be accountable to their audience? And while classes in malware writing raise a different set of problems, shouldn’t the institutions in question be accountable both to their students and to society as a whole?
Pingback: Security Curve: Right to Reply « amtso
http://www.risc-corp.com David Schneier

Conflict of interest be damned, but if not the vendors participating and providing resources than who? Just about every good (and popular) standard I can think of in my domain has had substantial involvement from the very firms that stood to benefit the most from their creation and adaption.

I understand your fundamental issue with it but how far do you think any such standard would get if the vendors didn’t endorse it? And do you think they would endorse it unless it was something they felt they could live with?
David Harley

Well, there is an element of conflict of interest, though I think that can be overstated, since (a) the testers in AMTSO are quite capable of holding their own against the vendors in the group (b) it’s the researchers who get sent to AMTSO, rather than the marketers. We have a lot of experience of cooperating for “the common good” if that doesn’t sound too cheesy, even though we work for companies in competition. In this case, the presence of multiple vendors actually tends to reduce conflict because we can spot a pitch for unfair advantage a mile off. More the Prisoners’ Dilemma than a pure zero-sum game.

But I agree: if you try to enforce testing standards without taking into account the underlying technology under test, you aren’t likely to get good testing, so you need the input of the people who create and maintain the technology.

By the way, if I can address a point I missed before: of course we should all be aiming for unbiased testing. Repeatable testing is far more problematic, because of the current nature of the threatscape. Reproducibility kind of assumes a static test set, but such sets are decreasing in validity in a world of short-life threats and serverside polymorphism.

Dynamic testing is a river, not a pond. You can’t step into it twice. So you have to look for other test validation strategies, such as more longitudinal testing.

AMTSO isn’t really about policing testers – it doesn’t have the resources to do that. It’s more about defining that kind of problem and looking for alternative methodologies. IMHO….