Why’s Everybody Pissed at Consumer Reports?

Posted by Ed in Analysis on Aug 17, 2006

Consumer Reports has apparently decided to test the capability of antivirus software to detect and respond to new and arising threats. In order to do this, they have contracted with an outside firm to create new malware which will then be scanned by the AV software. This sounded like a good idea to me, but then I read the reaction from the AV community:

[Sophos:] When I read about what ConsumerReports has done I want to bash my head against a brick wall. With over 185,000 viruses in existence was it really necessary for this magazine to create 5,000 more? It’s a bit like Fire Monthly Magazine testing fire stations by lighting umpteen fires around the country and seeing who is the fastest at putting them out. It’s irresponsible behaviour, and will be frowned upon by the anti-virus industry. Leave anti-virus testing to the independent testing bodies with expertise in the field.

[Kaspersky:] After all there are many many thousands of viruses in existence already and we’re adding around 200 new signatures to our database every day, why the need for someone to create new ones?

And so on. Everybody’s all in a tizzy about it. The AV folks claim that creating malware is wrong – no matter what the circumstances. The argument is that there is so much malware already that adding new malware to the list – no matter what the reason – is unethical. Now, maybe I’m an irresponsible lout, but
I think that’s USDA prime "bull". Why? Because #1 I don’t
accept that AV companies are the last stop when it comes to malware ethics and
#2 I think Consumer Reports is performing a useful service to the
community. In other words, I think it’s useful for customers to be able to quantify the efficacy of claims made by AV software vendors
with respect to detection of new malware – and believe me the claims in this
area are pretty big:

Norton AntiVirus (NAV) has the ability to detect unknown viruses of
various types using heuristic algorithms known as Bloodhound. [Symantec]
With advanced heuristics and generic detection it finds even new,
unknown viruses, even hidden in compressed files. [McAfee]
Sophos AV does incorporate heuristic scanning for unknown viruses in
the wild. [Sophos]

And so on. They all make the claim. How can we know which
work and which don’t. In order to test the reality of these claims,
consumer reports decided to create some new malware for these products to find. Why is that so wrong?
Let’s break down the objections one by one:

Objection #1: It’s wrong because the malware could get into the wrong
hands and tear a swath of destruction across the land. So, it
seems to me like we don’t know from what CR has said if the malware they
created had functional propagation capability or payload; we also don’t know
if it was created inside a safe and controlled environment. Is it OK
if there is no destruction, or possibility of destruction?
Objection #2: Because it means that AV companies need to write new
signatures. Um… No offense, but "cry me a
river". Look, AV companies are not a public service. As
part of their risk/reward analysis, these companies have decided that it’s
more cost-effective at this time to write new signatures when new malware
comes out vs. advancing the heuristic capability to the point where they
don’t have to. They went into it with their eyes open, and I’m not
about to agree that legitimate, useful research should stop because it hits
Symantec’s bottom line. Not in this lifetime anyway.
Objection #3: It’s wrong "no matter what the circumstances"
and "for any purpose". This is what I call the "lalala"
argument – remember when you were a kid and you’d put your hands over your
ears and go "lalala"? Yeah, that’s this. Basically, in this
view, it doesn’t matter why you’re writing it, what the payload/propagation
is, or what the effect will be – it’s just wrong. Since this argument
isn’t predicated on anything concrete or specific (i.e. "it’s wrong
because I say it is"), it’s somewhat hard to refute. However, I
think it’s useful to point out that since in this scenario it’s equally
unethical no matter how inert the malware is, that this means the minute
that you call something a virus it becomes problematic (for example if I
started calling Microsoft Word "Win32.OfficeProductivity.A" it
would then be unethical for me to have it.)

Well, I guess I went on about this one… It’s just one of those things
that gets me fired up.

http://riskmanagementinsight.com/riskanalysis/?p=11 RiskAnalys.is

On Anti-Virus

Virus, from the art of Marianne Lovink
The best post this morning comes from Security Curve Weblog on Consumer Reports vs. the Anti-Virus companies. I can see a FAIR-based scientific use for this sort of research, so I’m with CR (in addition to…
http://anti-virus-rants.blogspot.com kurt wismer

“#1 I don’t accept that AV companies are the last stop when it comes to malware ethics and #2 I think Consumer Reports is performing a useful service to the community. In other words, I think it’s useful for customers to be able to quantify the efficacy of claims made by AV software vendors with respect to detection of new malware”

1) it’s not just the anti-virus companies but the av community as well…
2) it’s possible to measure what consumer reports wanted to measure WITHOUT creating new malware and if they’d done their homework they would have found that multiple independant testing organizations already perform this measurement…

as per your objections:
objection 1) most viruses actually have no payload – do you really think that makes them ‘OK’?
objection 2) of course they have to write new signatures – there will never come a time when heuristics are able to completely obviate the need for signatures of one kind or another (unless you’ve found some clever solution to the halting problem)…
objection 3) the “it’s wrong for any purpose” argument comes into play because believe it or not over the past 20+ years the question of “is it ok to make viruses for this purpose” has been asked and answered to death and there has never been a scenario postulated in which the creation of new viruses was the best course of action or even necessary… it’s not a lalala argument, it’s just that most people don’t feel like rehashing this issue each time it comes up…
http://www.securitycurve.com/blog Ed

Kurt,

First of all, great comments! You really got me thinking about this…

So, I guess my first question is a philosophical one: what constitutes malware? For example, if consumer reports created a bunch of random files with random data in them (clearly *not* malware) and used them to test AV software, nobody would object, right? If they modified real malware enough to invalidate the signatures and also disabled the payload and propagation features, is that “safe enough” for them to test? I would argue that it probably is. What if they did create new malware *with* a destructive payload and viable propagation, but tested it under circumstances so controlled that it couldn’t possibly leak out? Maybe that’s safe enough, too. Or maybe it isn’t.

My second question goes to what constitutes an “independent lab” – why is consumer reports less qualified to test AV software than other labs? Maybe their procedures suck and it is unsafe, or maybe they’re better than the independent labs; we don’t know. But what *would* make it OK for them to test? Can they get certified from an independent body to be a qualified lab? Is there a clearly-defined list of procedures that they can follow to make sure they are doing the testing in a safe manner? Clearly, there are “best practices” and “recommendations”, but right now anybody can test AV (heck, Security Curve has even done it.) Since consumer reports invested the time and expense to do the testing they did, it seems to me that they have a legitimate business need to do so. Until there is some sort of central oversight requiring them to take specific measures, I’m not sure we (or anyone else) has the right to say that they can’t do it.

As to the halting problem, your point is well taken and I agree with it. I’m not sure that signatures will ever go away – it’d be nice if they would, but maybe they won’t. On the other hand, writing signatures is a current cost of AV vendors doing business – they can elect to include the CR new malware in their signature files or not, clearly there is a burden for them in terms of signature-authorship. But should we, as a matter of course, look to extra costs to the vendors as a reason to not do research? For example, should we not independently perform automobile impact testing because auto-makers might lose money based on what they find or how they do the test? Now, one could argue that these aren’t the same thing (and they’d be right), but again – where’s the line?

All I’m saying is that until somebody makes formal rules about how this testing gets done, I don’t think we can criticize particular firms (who ultimately have the best interests of consumers at heart) for not following those rules.

Again, just my two cents.

-E
albatross

Handing AV companies a list of 200 new signatures from specially-made malware is like spitting in the ocean, in terms of the existing number of known pieces of malware.

Can someone point out a link to a strong argument for why creating new malware isn’t an appropriate way to evaluate this kind of defense. I would assume in most other security scenarios that this was an excellent way to see if the security mechanism was working. For example, one way for the FAA to check whether the TSA’s screening procedures work is surely to come up with new kinds of bombs and see if mock-ups of them can be slipped onto planes.
http://allan.friedmans.org Allan Friedman

Out of curiousity, do the people who are loudly condemning CR tend of have a strong opinion on the question of public vulnerability disclosure?
http://anti-virus-rants.blogspot.com kurt wismer

@ed
“what constitutes malware?”

there’s no formal definition, malware is an umbrella term for any software that is considered bad/malicious… the malware in this case (as i understand it) is modified samples of existing viruses… whether that’s ‘safe enough’ is unknowable – the new variants must still be viruses (otherwise their test is invalid as they aren’t measuring what they set out to measure) but we have no idea how good the people handling those viruses are at handling them safely, which is where the real risk comes from…

“why is consumer reports less qualified to test AV software than other labs?”

ask yourself what properties would make any organization qualified… i think somewhere in there you’ll find that a certain degree of expertise is required and CR’s apparent ignorance of the av community’s mores with respect to creating malware and the fact that they contracted out the design of the test protocols makes me believe they lack that expertise…

“Until there is some sort of central oversight requiring them to take specific measures, I’m not sure we (or anyone else) has the right to say that they can’t do it.”

it’s not about being authorized to do it or not… they acted irresponsibly by creating malware (something we don’t need more of and would rather have less of) when it wasn’t necessary…

“But should we, as a matter of course, look to extra costs to the vendors as a reason to not do research?”

it’s not just vendor cost… creating new malware has a cost to society at large… you’re creating new risks for the computer using public… does the research justify doing that? how often and to what extent? CR increased the number of risks by a significant amount – are they going to do that every year? are we really better off for them doing that unnecessarily?

“For example, should we not independently perform automobile impact testing because auto-makers might lose money based on what they find or how they do the test?”

i don’t believe anyone suggested not testing av software at all, just that it should be done in a more responsible way… it’s possible to test how well products do at heuristic detection without creating new viruses and it’s clear that CR (and the people they hired to help them) didn’t do their homework…

@albatross
“Handing AV companies a list of 200 new signatures from specially-made malware is like spitting in the ocean”

it’s not 200, it’s 5,500 which is about a 3% increase in the number of viruses… if everyone who wanted to test av products did this every year we’d have a very big problem…

@allan
“Out of curiousity, do the people who are loudly condemning CR tend of have a strong opinion on the question of public vulnerability disclosure?”

it’s mostly the av industry and community condemning CR… i can’t say for sure whether there is concensus among them on vulnerability disclosure, but i’m pretty sure there’s concensus that vulnerability disclosure and malware disclosure are not comparable…
Safely anonymous

Kurt,

Frankly, I’d be a lot more impressed with the results of the “long-rehashed question” had that hashing not gone on in closed rooms, filled with a closed community. I’d be even more impressed if I could buy a Windows box, put AV software on it, and expect that AV software would actually protect me. But, really, I don’t. I think the AV industry is succeeding at what customers want it to do.

I think the industry is making lots of money not succeeding, and so is unlikely to change, and it seems to me that they’re all upset at Consumer Reports for pointing this out.
http://www.securitycurve.com/blog Ed

Safely Anonymous,

Wow! Double Wow! My hat is off to your cynicism.
http://anti-virus-rants.blogspot.com kurt wismer

@safely anonymous
“Frankly, I’d be a lot more impressed with the results of the “long-rehashed question” had that hashing not gone on in closed rooms, filled with a closed community.”

and what makes you think it was or is a closed community? as i was able to watch and even participate in some of it without any official credentials (in fact i started when i was still a teenager), i can assure you it was not all done in closed rooms or with a closed community…

“I’d be even more impressed if I could buy a Windows box, put AV software on it, and expect that AV software would actually protect me.”

i’ll be blunt… the kind of install-and-forget security you’re alluding to here is 100% snake-oil… it does not exist and can not exist and it’s a real shame that the mass media (and marketing departments) have managed to ingrain that fantasy so deeply into the public psyche that even intelligent people become indignant when it fails to come true…

“I think the industry is making lots of money not succeeding, and so is unlikely to change,”

they fail because of the technical constraints on a priori algorithmic detection…

“and it seems to me that they’re all upset at Consumer Reports for pointing this out.”

[sarcasm]
because manufacturing new hazards needlessly couldn’t possibly be considered a bad thing…
[/sarcasm]
Safely anonymous

Kurt,

“and what makes you think it was or is a closed community?” My attempts to participate in it; their continued and persistent insistance that distributing samples is bad and irresponsible, despite the reality that I get samples every morning in email.

“100% snake-oil” Having read Fred’s paper, I understand what you’re saying, but hueristics could be a lot more effective than they are now. Further, that’s not the marketing message any of these companies are sending.

“technical constraints on a priori algorithmic detection…” Sure. Taking in ~5 billion a year, they could invent some new methods.

“because manufacturing new hazards needlessly couldn’t possibly be considered a bad thing…”

that something could be considered a bad thing doesn’t make it a bad thing. given toolkits for construction, and given that they were responsible in how they constructed and tested, the industry’s abject horror of “something could go wrong” is childish.

What went wrong is that consumer reports tested AV products in a way that those products are tested every day, and found them wanting. Now, if their methods are flawed (and they are) then we ought to be seeing innovation in methodologies, and sometimes that requires some tipping over of apple carts.
http://www.securitycurve.com/ Ed

To chime in on a general point, I don’t agree that signatures are the only way to do malware protection; it’s possible that they are the only way to do malware *scanning* (due, as Kurt indicated, to the undecidability of the halting problem) – but it seems to me that scanning and protection are different things. Hypothetically, you could build a product around preventing malware symptoms rather than scanning for known patterns/strings… For example, if you wanted to prevent malware from overwriting the file system, you could use a read-only file system – you don’t have to analyze the malware at all to enforce the constraint. In fact, I think that the industry will ultimately have to change direction on scanning, due to peformance limitations of current scanning techniques (I won’t rehash it again, but linear search has well understood properties that don’t look promising for the future of signature-based scanning.)

In terms of whether or not to create malware, if we use the physical world as an analogy, there are plenty of labs that create new microbes: and some of those labs actually traffic in, manipulate, and share pathogens amongst each other. What makes this “OK” is the fact that there is oversight over who has the authority to distribute say, anthrax. Not just any old lab can create new strains and none of those labs can release new pathogens into the wild – ever. As a consequence of this oversight, there are no questions about who can or can’t do this kind of work. We don’t have that type of control with malware. Why not? Maybe because the consequences aren’t as huge… Maybe because it’s a younger science… Whatever the reason, this debate will continue until there are some rules in place – formal rules – governing how this happens.

Again, just my humble opinion.
http://anti-virus-rants.blogspot.com kurt wismer

@safely anonymous
“”and what makes you think it was or is a closed community?” My attempts to participate in it; their continued and persistent insistance that distributing samples is bad and irresponsible, despite the reality that I get samples every morning in email”

it’s easy enough to find garbage on the ground, that doesn’t mean it’s ok to add to it… the reason it’s bad is that it makes you part of the problem rather than part of the solution…

“”100% snake-oil” Having read Fred’s paper, I understand what you’re saying, but hueristics could be a lot more effective than they are now.”

that depends entirely on how you define effective… historically, dialing up the heuristic sensitivity does catch more things but it also increases the false alarm rate… the 2 most likely reactions to which are needless fear (when they don’t understand what false alarms are) or unwarranted confidence (when they do understand what false alarms are and assume the alarm is false because they’re too lazy to do what’s necessarty to resolve the ambiguity for each and every alert issued by the product)…

“Further, that’s not the marketing message any of these companies are sending.”

isn’t it? are the products not referred to as ‘solutions’ in spite of the fact that they don’t actually ‘solve’ the problem? do they not claim to protect the user just as you’re asking them to actually do in spite of the fact that they’re really just tools that can help the user protect him/herself?

“technical constraints on a priori algorithmic detection…” Sure. Taking in ~5 billion a year, they could invent some new methods.

[sarcasm]
yeah, sure, because throwing money at a problem is all that should be needed to solve it…
[/sarcasm]

the virus problem and the broader malware problem are not purely technological problems and they cannot be solved by technological means…

“that something could be considered a bad thing doesn’t make it a bad thing.”

do you think manufacturing new hazards needlessly is not a bad thing? if so i would be forced to acknowledge that we are likely never going to agree because our values are just too different…

@ed
“I don’t agree that signatures are the only way to do malware protection;”

you’re absolutely right, it’s not… signatures are just one possible method… it has it’s strengths and it’s weaknesses, as do all methods… ideally it should be used in conjunction with other methods that are able to complement it, ones that can be strong where it is weak…

“Hypothetically, you could build a product around preventing malware symptoms rather than scanning for known patterns/strings… For example, if you wanted to prevent malware from overwriting the file system, you could use a read-only file system – you don’t have to analyze the malware at all to enforce the constraint.”

i think in more general terms what you’re getting at is called behaviour blocking… it too has it’s strengths, but it also has it’s weaknesses and one of those weaknesses is that it allows the malware to run… if you can detect the malware before it gets control (as known malware scanning is able to do for known malware) then there is no opportunity for the malware to detect/disable/attack/bypass the protective mechanism… behaviour-based protection is fundamentally unable to stop malware before the malware is run because the malware’s behaviour only comes into play after it gains control… behaviour-based systems can be useful, but i wouldn’t use them in isolation – like known malware scanning, they’re best combined with other techniques (in fact, known malware scanning and behaviour based systems can do a lot to complement each other)…

i prefer to look at addressing malware by breaking it down into 3 parts – prevention, detection of preventative failures, and recovery from preventative failures… known malware scanning is quite good at prevention of known malware (and since the vast majority of malware happens to fall under that category, that makes known malware scanning pretty effective)… known malware scanning is essentially a blacklist technique but there are also whitelist techniques which are also good for prevention… i would classify behaviour based methods as detection of preventative failures (since they allow the malware to run)… behaviour blockers and change detectors and a number of other techniques fall into this category… backups, general purpose disinfectors (those usually found in known malware scanners), and dedicated malware removal tools (usually one-offs) are of course examples of methods that are meant to address recovery…

“What makes this “OK” is the fact that there is oversight over who has the authority to distribute say, anthrax. Not just any old lab can create new strains and none of those labs can release new pathogens into the wild – ever. As a consequence of this oversight, there are no questions about who can or can’t do this kind of work.”

actually there are still questions (surely you’ve heard people ask why does the military need to make new diseases, or ask whether it’s ethical, etc.), it’s just that the presence of that oversight makes most people feel more at ease about the whole thing… so long as it’s an authority figure and not just some guy somewhere most will assume it’ll be ok (and usually they’re right)…

“Whatever the reason, this debate will continue until there are some rules in place – formal rules – governing how this happens.”

and there’s the rub – to govern this implies involvment of the government… in reality there is no one, no group capable of exercising this kind of authority… it’s similar to trying to make local laws apply to the entire internet… it doesn’t work… maybe a country could put rules in place that affect testing organizations within that country, but there are testing organizations in multiple countries, and some people carry out their own tests as individuals which makes them nearly impossible to govern…
http://asteriod.divnull.com Asteroid

“the reason it’s bad is that it makes you part of the problem rather than part of the solution…”

In other words, “lalalala”.
http://anti-virus-rants.blogspot.com kurt wismer

@asteroid
“”the reason it’s bad is that it makes you part of the problem rather than part of the solution…”

In other words, “lalalala”.”

since my argument about why distribution is wrong regardless of how easy it already is to find malware consisted of more than just that one statement, i guess the above is your own way of saying “lalalala”…

i suppose i could have been more obvious and said 2 wrongs don’t make a right, or if everyone else jumped off a bridge would you do it too, but i chose to go with the litter analogy…
http://www.emergentchaos.com Adam

That bridge argument stopped having validity when some bozo invented bungee jumping.
http://anti-virus-rants.blogspot.com kurt wismer

@adam
“That bridge argument stopped having validity when some bozo invented bungee jumping.”

perhaps, but bungee jumping doesn’t do anything for the litter argument, does it…
http://www.securitycurve.com/blog/archives/000475.html Security Curve Weblog

AV Vendors need to crank it down about a million degrees

I came across an article today about John Aycock and his new spyware class at the University of Calgary. Dr. Aycock is of the opinion that students learn better how to protect against spyware by first understanding how spyware works – and what better w…
http://www.pissedconsumer.com Pissed Consumer – Jennifer

creating malware is wrong .. period…
http://www.securitycurve.com/ Ed

Jennifer,

Well, your argument is the same as quite a few other people. My question though is, “why is it wrong… period?” Is it because that malware could get into the wrong hands? If so, who decides who the right hands are. Is it wrong because only AV companies are allowed to create malware? Who says? Do they have to get a special license or something? Not currently.

My issue is this: I don’t trust Symantec (or McAfee or Sophos) to say who can or who can’t write (or test) malware. So when they say that they are more qualified to do this than Consumer Reports or the University of Calgary (Google for “Dr. John Aycock” to see how that went down), I question who died and made them king of the forest.

-E