OK. So maybe I was wrong…

I
really don’t like admitting when I’m wrong.  I really, really don’t. 
But looking down the road and imagining what the world would be like based on
some of the recent developments in the application security space gives me
pause.  I think it’s time that I stop and rethink some of the premises that
I’ve held to be true and reexamine the traditional thinking that is  widely
held in the security community.  Let’s play a few "thought
exercises" based on recent events and I think you’ll see what I
mean…  

Scenario 1: Imagine you’re a maintenance programmer working at a typical development
shop.  Maybe you’re working at a large software shop or maybe you’re at a
smaller vendor; either way it doesn’t really matter – the key fact is that your
job is to fix bugs.   When a security-related bug comes down the
pipeline, your job is to find the source of the bug and fix it as fast as
possible – you know that your customers, your boss, marketing, your
shareholders, and maybe even the press are all waiting for you to get that patch
out as soon as possible.  Maybe you need to work through the night tracing
down some difficult problem in order to debug and publish the fix.  What
happens if while you’re working on one bug, another shows up in your
queue.  You sigh, put it on the "back burner" and continue with
what you were doing.  Then another comes in.  Then another.  And
another.  And another.  Your queue starts filling up.  Your
time-to-complete goes from days to weeks to months in minutes.  No time to
stop and plan, no time to hire help, no time to share the load.  You’ve
lost… Game over.  

Scenario 2:  This time, you’re a sys-admin.  Your job is to
keep the desktops and servers in your environment patched and to make sure that
you implement workarounds for new security bugs as they’re found.  One day,
you walk into the office and find that you have a hundred advisories with
instructions about tasks that you have to perform to keep your machines in safe
working order.  What do you do?  A conservative estimate would be that
it’ll take you 2 months to install all the patches and various
workarounds.  Where do you start?  Do you just "start at the
beginning" and work through?  Do you prioritize the machines based on
criticality (maybe you’ve already done that), spend a month or so
regression-testing your apps, and then apply the patches?  How can you
possibly keep up?  Answer: you can’t.

Some of you are thinking that these scenarios are pretty far-fetched, am I
right?  After all, it’d have to be a pretty fantastical situation to arise
for a veritable "wall of bugs" to overload the traditional "find
and fix" model, right?  After all, we’ve had the traditional
"find and fix" model for years now, and everything’s been fairly
copasetic up until now.  Vulnerability researchers find a few bugs a month,
vendors fix them without taxing development resources and system administrators
deploy them without impacting their workflow over-much.  So why
worry?  Well, guess what: the fantastical is here.  

Now, let me start my critique of the current situation by saying that I love
the Metasploit Project; I use their framework professionally and I think what
they’re doing for the security community is top-flight.  I’m behind them
all the way.  However, recently, to raise awareness about a vulnerability
research technique called "fuzzing", HD Moore published a "bug
a day" throughout July; as a result, his purpose was served – I (and I’m sure I’m not alone) am now
fully aware of the power of fuzzing as a vulnerability testing technique. 
Today, we find out that fuzzing has led to the discovery of more than 100
flaws in various ActiveX controls installed by default on Windows XP. 
Sure is quite a few.  So, under the current "find and fix, full
disclosure" model, maintenance programmers over at Microsoft are
experiencing our "hypothetical" scenario #1; these guys are already
working on the multiple browser bugs from last month and now they have 100 more
on their plate.  When the patches are ultimately released for this, sys-admins
in MSFT shops will be experiencing nightmare scenario #2.  Get ready,
’cause here it comes…

Now none of this is HD’s fault.  He’s a reputable guy and one of the
shining stars in infosec.  His tool does what it does – and it does it
really damn well.  However, I’m concerned about what the broader effect of
the new technique might be on our current "bug-fixing infrastructure." 
Am I alone in thinking that bad-guys are looking at his tool and drooling over
the possibilities of what can be done with it?  Something tells me they’re
drooling for it like Pavlov’s dogs.  How about researchers who care more
about getting attention than they do about enhancing security?  If they
found 100 bugs, might they be tempted to disclose them all en masse and
get a good story in the press out of it?  Seems like they
might.  

So, clearly there’s a problem.  The infrastructure that we have in place
for fixing bugs is not sufficient to handle this kind of bug-discovery pace and
I have yet to come across a company that has the kind of administrative
infrastructure to handle this kind of pace for patch release.  So what’s
the answer?  HD Moore proposes (rightly) that it behooves vendors to learn
about fuzzing and use it themselves.  Maybe they will.  But what about
in the meantime?  What about legacy software (there’s a ton of it out
there.)  What do we do?  Staff up? If you’re used to fixing 5-10 bugs
a month, how many maintenance programmers would you need to fix 100 bugs, or
1000 – how about 100000?  So just going by the ratio and ignoring economy
of scale – if you have 25 programmers fixing 10 bugs/month, wouldn’t you need
2500 to fix 1000/month?  That’s if just 10 researchers used the same
(freely available) fuzzer tool.  In salaries that’d be an increase in
overhead of $250 million dollars; to put that in perspective, that’s half
of Netflix’s projected revenue for 2005.  Is it worth it for software
companies to stay open in that world?  Something tells me that cost would
have a negative effect on the bottom line (understatement.)

So what’s the answer?  I’m not entirely sure.  However, I can tell
you that I’m thinking full-disclosure in a "fuzzed" environment isn’t
the same kind of tool that it was before it.  I’m a fan of full-disclosure,
but if we (as
Pete Lindstrom once jokingly recommended,) made bug-finding illegal, some of
the symptoms would go away.  I’m not sure that’s the right approach and I’m
not advocating it, but maybe it’s not so far off the mark as I once thought.