"Open-source projects certified as secure" – huh?

I really got a chuckle out of this news item, especially this line:

“Coverity, which creates automated source-code analysis tools, announced late Monday its first list of open-source projects that have been certified as free of security defects.”

So we finally have the security silver bullet!

Run this tool on your code, fix the bugs, and you’re secure (and maybe unbreakable?!) I don’t think so.

There are three big problems with this line of thought:

1. First, the security bugs found are only the security bugs found by the tool, and that list is always smaller than the list of all bugs.
2. Second, it assumes that any new code or code changes are bug free. Which may or may not be true. In my experience, it is rarely true that new code is utterly bug free if you don’t take a holistic, process-oriented view to security.
3. Third, and this is probably the most important, at best the tool understands a subset of today’s vulnerabilities; that could all change tomorrow when a new class of vulnerability or a subtle variant is found.

The last point is important; security is a constantly evolving environment, and that's why we update the SDL regularly, to improve the process as we learn of new threats and design new defenses and mitigations.

Tools are very useful, we build a lot of tools, and use them all the time here at Microsoft. Some of those tools have found their way into our SDKs and Visual Studio so our customers can use them too. But I would never claim that these tools make code "free of security defects."

Comments

• Anonymous
  January 10, 2008
  Come on, you are just jealous because their code is d3efect free and your isn't!

• Anonymous
  January 10, 2008
  The comment has been removed

• Anonymous
  January 10, 2008
  The comment has been removed

• Anonymous
  January 10, 2008
  Coverity makes Static Analysis tools.  They ran their static analysis tools on some OSS projects, and the projects had little/no defects.  I can buy that. However, as we all know (or should know), static analysis tools do not catch all security vulnerabilities.  They are just one piece of the puzzle. It's great that they are doing this.  Code from the OSS projects listed are used in a LOT of embedded devices.  I think the PR department of Coverity and News.Com are oversimplifying things a bit, though.

• Anonymous
  January 10, 2008
  Michael, while your description of SDL is fair (and balanced), you might have a look at the original press release at Coverity's website before giving such a harsh judgement. Their approach is much more sophisticated than you insinuate, they are well aware of their limitations (as you are with SDL). Regards, Thomas  

• Anonymous
  January 11, 2008
  I got a big kick out of that series of articles as well.  Any legitimate software security practitioner is laughing, but the scary thing is that folks who don't understand software - which includes a lot of traditional information security people - may take this at face value. With Coverity saying that those projects are "certified as free of security defects" they may as well say they're ... Unbreakable :) That worked out well for those other guys, didn't it? --Dan

• Anonymous
  January 11, 2008
  This is one of the things that scares me the most when I wrap up a code review project and the security bugs found have been fixed. As much as I say over and over this is only saying we fixed the ones we found and there may be ones we didn't find or a new type problem may surface tomorrow that we weren't looking for, I know some will think the product is perfectly safe. No need to keep looking. No need to check the new versions. No need for more training of the developers. So far that hasn't actually happened but it's a nagging fear always in the back of my mind.

• Anonymous
  January 11, 2008
  Thomas - my comment is about the press article!

• Anonymous
  January 11, 2008
  The comment has been removed

• Anonymous
  January 11, 2008
  PHP is in the list, this should be a cruel joke! Especially after Month of PHP bugs :)

• Anonymous
  January 11, 2008
  The comment has been removed

• Anonymous
  January 21, 2008
  FOSS is used a lot (like a guy said, tons of embedded devices use OS and while at it, why not search the MS TCP/IP stack of certain OS revisions for the string "BSD"?). While a static analysis tool can only catch the most rudimentary of errors and yes, Coverity is using this for their PR campaign, even if just a few bugs are found (and fixed!) this is good news for all of us. I will not go into the whole PCI compliance thing with i's to dot and t's to cross, but come on, fewer bugs is good news!

• Anonymous
  April 01, 2008
  Just found this today, April 1 2008, there is a new scanless PCI option being offered for free in addition to the traditional non-scanless options at scanlesspci.com "We believe the certification process should never interfere with your ability to do business, nor be nothing more than an excuse to sell you unnecessary products and services. We've pared down PCI compliance to it's essence, offering industry-standard certification, and nothing more."