Comment by _benj
3 years ago
I once went through a “security audit” and I couldn’t help but feeling like all they did was run my site trough ZAP’s automated tests.
Even the links and categorization they used where strikingly similar to ZAP’s output.
Now, not everybody knows about ZAP and they might benefit from such assessments, it also happens that the higher ups tend to not trust their people and seeing a spreadsheet with a logo from a security company is comforting to them, but at any rate, I found no value at all whatsoever from the experience except from the commendation of the CEO about our good security practices since nothing major showed up on the audit (I had taken a cursory look at bed practices and used common sense like defaulting to distrust user input)
All I’m trying to say is, using a tool like ZAP (https://www.zaproxy.org/), which also happens to be an amazing tool for development too usually can take you very far with its automated testing, and if the higher ups still need a rubber stamp from a third party, well, that’s up to them…
I had nmap screenshots presented as a penetration test. I don't mean "in a penetration test, with some text". I mean a penetration testing company embarked on a two week engagement to review an application I had built, and they literally handed in a screenshot of nmap on their own letterhead and called it the report. I was pretty livid.. there a lot of shortcuts on security I was actually hoping to get a drive to improve, but instead I got hauled to "please explain" what this "port 443 is open" report means.
Oh wow! I’d be pretty annoyed too!
There's a mix of naming here. The original post was about soc which is more about organisation posture than specific apps. What you got was a crappy pen test. What you were advertised was an audit. I don't have a solution unfortunately apart from posting here to let more people know they're different things and they should do research and set expectations of what results they want before dropping big $$$.
Thanks for the clarification, my bad.
> and if the higher ups still need a rubber stamp from a third party, well, that’s up to them…
It is a borderline racket where everyone is insisting on this cert from everyone else they are doing business with at the enterprise level.
Once got to read a “penetration test report” that a past employer received and paid $30,000 for after badgering the Infosec Director to see it.
The “methodology” was as comprehensive and revealing as a ping test. The doc contained a single screenshot, was riddled with both grammatical and technical errors about the environment they were supposed to be ‘penetrating’.
Made me wonder if I’m in the wrong segment of tech if this is what companies are throwing away for such shitty tests when I know I could do a better job.
On a previous role I decided to open ZAP for a quick debugging, but during it I decided to just edit the user id that was being sent on my request, and to my horror our app sent sent me all the data for whatever user id I would send!
This was a project that had millions of dollars in man hours and where many developers with a lot more experience than me had worked for years…
Bottom line, they had a monkey patch custom framework that wad just Django without documentation and with huge security holes…
I sometimes wonder if that industry exists because if suits don’t see a $30k to fix checking an user id, they couldn’t be bothered to have engineers do “useless” tickets to pen test their own systems
Burp is a lot better than Zap, but costs money. If a company is charging thousands for a test but isn't paying for Burp, that sounds like a red flag.
I think that could be a good question to ask a security company before hiring them, what tools do they use.
Pentests are really different from soc2 type audits.
But quality of pentests do vary significantly depending on who you buy from.
Rubber stamps can be useful as an absolute minimum base line, because companies will lie through their teeth that their product is "secure", and at least with even a poor pentest you know you probably wont get easily hacked by a script kiddie, which may be a valuable assurance.
When you get easily hacked by a script kiddie, you can CYA internally about all the “preventative measures” you took by showing off the rubber stamp.