On penetration testing – harmful?

Over at Sensepost Security, there’s a new blog entry wondering about Haroon Meer‘s talk “Penetration Testing Considered Harmful“. Those who know me know that I’ve had this view for a very long time. I’m sure you could find a few posts in this blog.

Security has to be a intrinsic element of every system, or else it will be insecure. Penetration testing as a sole activity and piece of assurance evidence makes security appear on the fringes of the development, something that you pass or fail, something to be commodotized, a box to be ticked, and ultimately ignored. Penetration testing as is done by most in our industry is incredibly harmful. It’s a waste of investment to most organizations, and they know it so they try to minimize wastage by minimizing the scope, the time, and poo-pooing the outcomes.

Penetration testing should be a part of a wider set of security activities, a verification of all that came before. All too often, we come across clients who want to do a one or two day test the day before go-live. They’ve done nothing else, and when you completely pwn them, they’re terribly surprised and upset.

We need to move on to make penetration testing the same as unit testing – a core part of the overall software engineering of every application.

Penetration testing should never be ill informed (zero knowledge tests are harmful and a WAFTAM for all concerned), and it should have access to source, the project, and all documentation. Otherwise, you’re wasting the client’s money up against the wall and acting unethically in my view.

Tests should come from the risk register maintained by the project (you do have one of those, right?), as well as the use cases (the little cards on the wall) as well as from the OWASP ASVS / Testing Guides. More focus must be made on access control testing and business logic testing.

Penetration testing has become vulnerability assessment – run a tool, drool, re-write the tool’s results into a report, deliver. No! Write selenium tasks and automate it. If you’re not automating your pentests, how can your customers repeat your work? Test for it? They should be taught how to do it.

Folks at consultancies will shriek away in horror at my suggestion, but getting embedded is actually a good thing. Instead of hearing from a client once in a blue moon, you’re integrated into the birth and growth of software. This is a huge win for our clients and the overall security of software.