More about vendor research

“He uses statistics as a drunken man uses lampposts – for support rather than for illumination.” 
                                                                                       ~Andrew Lang 

So, I feel like I’m on a one-man crusade sometimes.  Today, I came across
(via HackInTheBox)
a TechWorld article
called "Web Apps the Number One Security Blindspot" which basically
states that applications are a security "black hole" and that they’re
constantly being attacked with none to notice.  The article draws on a recent
report from
Fortify where they sampled a number of sites looking for attack patterns in
the wild and drew a number of conclusions based on those findings. They point
out that there’s a ton of activity going on out there in terms of application
attacks, and the further extrapolate relative prevalence of various attacks as a
percentage of overall attacks.   The instrument that they used to
collect the data of course, was their for-profit commercial tool.  

Now, I’ve pointed this out before, but there’s an inherent problem with
vendors producing research like this; particularly when that research uses their
commercial tool as the detection instrument. Specifically, these vendors
typically have a niche – and the reports produced within that niche are only
reflective of one particular area of focus.  For example, Fortify’s report
doesn’t have anything about phishing activity, malware, fraud, etc.  Is
that to say that these things don’t happen?  Of course not.  It’s not
mentioned because Fortify doesn’t do fraud detection, AV scanning or
anti-phishing solutions.  If they did, I bet it’d be in the report. 
Instead, what’s in the report is only what’s caught by their product.  So,
when they say that "On average, 50%-70% of attacks experienced by web
applications come from bots and bot networks searching for known
vulnerabilities", what that really means is "On average, 50-70 percent
of the attacks that Fortify detects are from bots" – and that’s probably
because automated, consistently-formatted attacks are more likely for a scanning
product to reliably detect. Plus, having a vendor publish these things tends to
lead to semi-biased conclusions like, "Fortify