AMTSO response roundup

Posted by in Recaps on Jul 14, 2010

I just wanted to call out the excellent responses and furthering of the recent AMTSO discussion from Kurt Wismer here and Andrew Lee from the Avien blog here.  I really don’t have much to add to the discussion other than what I’ve already said about it, but I wanted to call out the well-written responses.

One of Kurt’s points was that the context he brought to the AMTSO NSS report review brought him to a different set of conclusions than I did.  That sounds reasonable – Kurt is more involved in this than I am (both the AMTSO work generally  and the technical underpinnings of anti-malware specifically).  However, I think these review documents are pretty important – and if they are written for the public, they should endeavor to be unambiguous.  If they’re not written for the public, they shouldn’t be accessible to the public.  I’ll get to why in a minute.

Andrew’s point is that I’m missing the boat on what AMTSO is trying to do (or that maybe AMTSO needs better PR.)    Well, I agree with the PR thing (I wasn’t sure if Andrew was kidding or not, but I’m serious about that), but the first part I do take issue with…  It’s not that I’m not on-board with what AMTSO’s up to or that I don’t think it’s valuable.  To the contrary, not only do I agree wholeheartedly, but the reason I’m so vocal about this is that I’d really like to see it succeed.  And right now, I think the goal is in jeopardy.  Why?  Because as it stands, there is opportunity for an irate malware vendor to use the AMTSO test review process as a weapon to discredit a test in the court of public opinion.  I’m not going to speculate about whether that’s what happened with the NSS test or not, but I will say that there’s nothing preventing it.  An unscrupulous vendor could try to challenge a test with no other rationale other than discrediting an independent test.

Now, I don’t think that anybody is evil over there – I think the folks working on this are very well intentioned.  But as anybody who’s watched Law and Order can tell you, sometimes it’s not about who’s right or wrong but about who can make the better argument.  Don’t underestimate what bakeoffs mean to product vendors as far as marketing goes – they can, and will, spend significant amounts of money to improve the product in specific ways to  increase their chances of doing well.

And if they don’t do well?  The temptation to discredit the test could be significant.  Could someone game the process to try to use this process for marketing purposes?  So long as the answer to that question is “yes”, I will advocate change.  If we’re setting up new processes to make things better, let’s do it as best we can.  And if we see an issue looming, let’s try to avoid it.

Search
  • http://www.avien.net/blog Andrew Lee

    I think the point about the challenge of an ‘irate vendor’ is worth commenting on. In fact, it requires two vendors to nominate a test for review (unless its testers asking for their own test to be reviewed, which needs no further nomination). Of course, we could equally argue then that those vendors can be in collusion. But then, the test is sent to the Review Analysis Board (RAB)(I serve on this board, so I’m fairly familiar with its workings) – and they will decide whether to accept the review or not. If they accept the review, then the RAB will form a Review Analysis Committee (RAC) from the AMTSO membership to conduct the review of the test. This committee may not include the original vendors who made the nomination – for precisely the reasons you state. The RAC then will do the review, and present their findings back to the board, who have the ultimate vote on whether the findings of the RAC will be accepted or sent back for further work. The only criteria on which the RAB will accept a review is if it has been clearly done in line with the AMTSO Guidelines, and the results of the RAC’s analysis are consistent with those guidelines. All of this is to try to ensure as far as possible that there is no individual bias in this, nor that a single vendor can simply ‘spank’ any tester based on a poor result. Not only that, but the rules for the review process explicitly state that “The procedures, including the member request, documentation and review process shall not be confidential, and all records regarding the review process and request shall be available to any member in good standing who makes a request to review such documentation. ” This means that any tester if they are a member of AMTSO can review for themselves the full process and documentation produced for that report by the RAC and RAB, including reasons for acceptance/rejection. The full documentation is here http://amtso.org/amtso—download—amtso-analysis-of-reviews-process.html
    And no… I wasn’t joking about the PR thing :)
    Thanks for your thoughts, they are appreciated. AMTSO is about openness and debate, even if the commentary is not always complimentary.

  • http://www.securitycurve.com Ed

    Andrew,

    Thank you for the comments. Very informative. I will read through the process and refine my opinion as appropriate…

    -E

TwitterRssFacebook