Tell HN: Information security audit / consulting is largely a scam industry
3 years ago
In light of a recent thread about SOC 2 certification (https://news.ycombinator.com/item?id=32018066), I wanted to share my perspective as an auditor/consultant on the other side of the table and inform people just how grim it looks from the inside. Before I get dunked on, yes - there are probably smaller niche firms worth every penny.
Shortly after starting in this line of work, it became clear that the services we sell are disingenuous. Here are some examples of why:
* My main argument: there is a HIGH likelihood that information security consulting is the first job out of college for the auditor leading you through the engagement. Beyond surface-level knowledge about multi-factor authentication being important and knowing that “Splunk is where the logs go,” your assessor is probably just nodding their head, asking canned questions from a spreadsheet, and not fully comprehending what you are telling them.
* We are told to describe ourselves as information security experts. I am not an expert. Every time I have to describe myself as the expert, I die a little inside. If I am the expert in the room yet still a recent college graduate, there is a glaring problem here.
* The middle manager of my department did not know the difference between a public and a private IP address when we reviewed DRL evidence together.
* The person leading your engagement may have a slight idea about what is going on, but they probably are tied to five other engagements and are not genuinely motivated to find problems because they are already underwater.
I can’t say that information security consulting is all bad. On several occasions, I have helped companies remove the clueless CEO from Domain Admins or explain why adding MFA to Cisco AnyConnect was a good idea for them. I should also mention that these types of positions are great for learning the inner workings of large companies that you might want to work at later on, how to passably write a report, and how to present information to executives.
Maybe I am preaching to the choir here. Interested to hear others' perspectives.
People in their twenties with zero courtroom experience do the vast majority of the work of the US Supreme Court.
People right out of college with zero experience help Fortune 500 companies with strategy, mergers, acquisitions, process design, enterprise system implementation, auditing their financial statements, etc.
The world is run by young people with minimal experience. Their organization's procedures, checklists, on-the-job training, and occasional guidance by someone with more experience somehow make it work.
And remember that your clients are even less knowledgeable than you and your checklist.
In the land of the blind, the one-eyed person is a god.
Pretty much all academic research work is done by grad students, with highly variable supervision by the PI.
Complex tasks often contain chunks which can be delegated from more to less qualified members of team (for mutual benefit). It's fine. But selling worktime of absolutely inexperienced people who at times don't even understand that spreadsheets they carry around as experts, and pretending there is some sort of superintellectual efforts were done when charging customers is still looks very much like snake oil business. And "security" industry is full of this stuff
Damn, imagine if there was some way to actually record all of this menial work that people do in some form of code, and then make that shit public.
As someone on the receiving end of dozens of these audits per year, it’s extremely obvious that both sides are just checking off their boxes and don’t have any actual understanding of anything. One side has to perform that audit, and the other side has to comply with it. Everything is outsourced to the lowest level people possible. It’s essentially a zombie industry where both sides just mindlessly generate documentation so the business people can check their own boxes before closing a deal.
Gives higher ups peace of mind. Same reason they love to move everything to the cloud and happily ignore any privacy concerns.
If you run something in-house and there is a security breach, it gets really uncomfortable, as you need to ask questions, assign blame, draw consequences. If $BIG_SAAS_PROVIDER gets their entire database leaked then well, it's not your fault, also if even $BIGCORP gets hacked it must have been really professional hackers and it can hardly be you made a bad call here. Nobody ever got fired for buying IBM.
But at the same time it kinda works to keep everyone in line. Look at PCI for example. You might be shit on the upkeep front, but ever X years some rando with a checklist will whip you into line while not really knowing anything at the same time..
I once went through a “security audit” and I couldn’t help but feeling like all they did was run my site trough ZAP’s automated tests.
Even the links and categorization they used where strikingly similar to ZAP’s output.
Now, not everybody knows about ZAP and they might benefit from such assessments, it also happens that the higher ups tend to not trust their people and seeing a spreadsheet with a logo from a security company is comforting to them, but at any rate, I found no value at all whatsoever from the experience except from the commendation of the CEO about our good security practices since nothing major showed up on the audit (I had taken a cursory look at bed practices and used common sense like defaulting to distrust user input)
All I’m trying to say is, using a tool like ZAP (https://www.zaproxy.org/), which also happens to be an amazing tool for development too usually can take you very far with its automated testing, and if the higher ups still need a rubber stamp from a third party, well, that’s up to them…
I had nmap screenshots presented as a penetration test. I don't mean "in a penetration test, with some text". I mean a penetration testing company embarked on a two week engagement to review an application I had built, and they literally handed in a screenshot of nmap on their own letterhead and called it the report. I was pretty livid.. there a lot of shortcuts on security I was actually hoping to get a drive to improve, but instead I got hauled to "please explain" what this "port 443 is open" report means.
Oh wow! I’d be pretty annoyed too!
There's a mix of naming here. The original post was about soc which is more about organisation posture than specific apps. What you got was a crappy pen test. What you were advertised was an audit. I don't have a solution unfortunately apart from posting here to let more people know they're different things and they should do research and set expectations of what results they want before dropping big $$$.
Thanks for the clarification, my bad.
> and if the higher ups still need a rubber stamp from a third party, well, that’s up to them…
It is a borderline racket where everyone is insisting on this cert from everyone else they are doing business with at the enterprise level.
Once got to read a “penetration test report” that a past employer received and paid $30,000 for after badgering the Infosec Director to see it.
The “methodology” was as comprehensive and revealing as a ping test. The doc contained a single screenshot, was riddled with both grammatical and technical errors about the environment they were supposed to be ‘penetrating’.
Made me wonder if I’m in the wrong segment of tech if this is what companies are throwing away for such shitty tests when I know I could do a better job.
On a previous role I decided to open ZAP for a quick debugging, but during it I decided to just edit the user id that was being sent on my request, and to my horror our app sent sent me all the data for whatever user id I would send!
This was a project that had millions of dollars in man hours and where many developers with a lot more experience than me had worked for years…
Bottom line, they had a monkey patch custom framework that wad just Django without documentation and with huge security holes…
I sometimes wonder if that industry exists because if suits don’t see a $30k to fix checking an user id, they couldn’t be bothered to have engineers do “useless” tickets to pen test their own systems
Burp is a lot better than Zap, but costs money. If a company is charging thousands for a test but isn't paying for Burp, that sounds like a red flag.
I think that could be a good question to ask a security company before hiring them, what tools do they use.
Pentests are really different from soc2 type audits.
But quality of pentests do vary significantly depending on who you buy from.
Rubber stamps can be useful as an absolute minimum base line, because companies will lie through their teeth that their product is "secure", and at least with even a poor pentest you know you probably wont get easily hacked by a script kiddie, which may be a valuable assurance.
When you get easily hacked by a script kiddie, you can CYA internally about all the “preventative measures” you took by showing off the rubber stamp.
As panarky said [0], the world is run by the young
They say "war is young man's game" - think about the 10s of millions of young men (some as young as 14) that went to war in WWI, WWII, etc
A group of 17-19 year olds is commanded by someone who's only 20 or 21
That you have any security experience, checklists to follow, senior people to rely on (ie the majors and staff sergeants, to continue the analogy) means you're worlds ahead of most people you're going to come in contact with
In Outliers, Gladwell claimed/popularized you need 10000 dedicated practice hours to be an "expert"
But in practice you only need about 200h to be in the top 10th or 5th percentile (or even higher) [1][2] (related: [3] & [4])
And regardless of how "experienced" someone else is (even how much more "experienced" they are than you), they don't have your team behind you, nor the focus that you bring
-------------
[0] https://www.flyingmag.com/what-makes-expert
In my opinion, most audits for security are the same as having an accountability buddy. The company goes I'm doing x, y, and z then the auditor collects and organizes the proof for or against those assertions. Like a good project manager, a good auditor doesn't have to be a domain expert. Security professionals tend to have strong opinions on what the right way to do a thing is, so an auditor with years of security experience can be a toss up whether it will be a help or hindrance. There are security consultants that provide significant value in reviewing and providing recommendations, but they can be difficult to find among the chaff. And even if the company hires an amazing consultant or security service, the company still has to adopt and implement. Information security at organizations is highly dependent on the executives and how they view security measures.
A good auditor will do exactly that: audit the evidence to certify compliance with the documented controls. It is not their job to specify controls, unless the controls are specifically called out in the standard against which they are auditing. They can flag controls that do not meet the standard, but they cannot specify what those controls must be.
However, a good auditor should also make recommendations on how to improve controls without holding the threat of a qualified report over your head if you don't comply.
So while I agree that they do not have to be a domain expert to do the bare minimum, such audits that they can provide are not especially useful in the long run. I want the value add of recommendations from a domain expert. This, however, is not that common.
Worse are the guys that think they have a badge. Their position has gone to their heads, and they are a nightmare to work with. I hate having to school these guys that it is not their job to demand specific controls (in the case of SOC2 for example), but to compare the controls in place to the standard, and verify that they are indeed in place. Unfortunately, all these guys work for accounting firms, and their bosses are generally clueless, so the "cyber guy" runs rampant. Some auditors need an audit.
Some people just want third-party assurance for CYA purposes. I’ve had to go through “security reviews” for apps submitted to third party platforms. One of them told me upgrade our web server because we had a vulnerability. That vuln was in some module that wasn’t even enabled. I explained this, and they still demanded we upgrade. So I just turned off the version identifier. Their scan passed. (I did upgrade nginx later anyway.)
Many security compliance scans/audits, especially at larger companies are about checking boxes and not practical security. I've done the same version hiding to get past nuisance reports.
Versions SHOULD be hidden from public queries. A proper assessment should identify versions anyway, by other means. If you hide them from your assessors to pass a scan, have Cyber insurance and later have an incident, coverage may be invalidated due to fraud (hiding data from assessor) or incompetence on the part of the assessor you hired.
Assessment teams have varying quality. Get the best results for your org you can out of them.
1 reply →
If the vulnerability is present it’s still a vulnerability. Just because the module isn’t enabled doesn’t mean it can’t still be taken advantage of in a remote code execution scenario.
Not using a library !== not vulnerable.
I forget the specifics, but there was no way to exercise the module remotely. I think it was actually Apache, not nginx, and the module was not even loaded. It was one of those bullshit "medium priority" line items.
5 replies →
The point of these types of audits isn't so much to determine "security" its to leave a paper trail so that people aren't accidentally lying to customers. It proves that you are either doing what you claim to be doing or intentionally being decitful. It cuts out the murkey middle ground, of "we didnt do what we claimed, but its all a misunderstanding & accidental so you can't blame us".
How much it suceedes is debatable but auditers being security experts is irrelavent to those goals.
Don't forget the cases where the companies are deliberately lying to customers. I worked at a company that pretty much lied to its customer's security teams about what they did. Not borderline, but claimed things they didn't do and didn't have plans to implement. This is what the SOC2 audit is for.
Two things: OP is SOC2'ing & also the mgmt of the firm seems inexperienced but trying to sell expert services.
Which is very different from pen testing, internal/external vulnerability assessments, PCI level 1/2, etc.
SOC2'ing is often like an auditor verifying that one does things according to a prescribed, or described process that the company attests to. I've often viewed it as 'accounting style firms' looking to get a bite of that 'juicy security scare fee pie'
"SOC'ing"??
My made up word for running a SOC 2 audit. Here is a link [just the first Google result] https://www.imperva.com/learn/data-security/soc-2-compliance...
Just because you’re not a security expert doesn’t mean security experts don’t exist.
And it certainly doesn’t mean security review is useless. You don’t even need to be particularly good to catch some of the worst things I’ve seen. Soc2 doesn’t necessarily mean you are secure but it does require the organization to do quite a few things that it’s a good idea to do. Periodic internal security review, having a password policy. There are few things in soc2 that are actually useless. Real security requires more than the compliance checklist, but the checklist isn’t a bad place to start.
Honestly deciding to care about security is a good place to start. Look at Colonial Pipeline. If they’d cared about security at all they could have hired a security engineer or even promoted a capable internal IT person to security engineer. That person could have even been wildly under qualified. Imagine going to work your first day as a security engineer, knowing you don’t know shit. What’s the first thing you do? Google security best practices. One of the top ten things is don’t share passwords. Another is don’t reuse passwords, another is rotate passwords when employees leave. (So write a password policy for the first two, but 1password, create a checklist Telford the third) Those three things would have prevented the thing that actually got them breached. The rest of the list of top ten would have prevented them getting ransomwared.
None of the bad security I’ve seen has been particularly hard to identify or fix. The problem has always been insufficient will to fix it.
I have yet to see an organization that implemented all of what I would call the easy stuff. I’d be thrilled to see it because it would mean we could dig into the truly hard stuff.
The problem is that even the "easy stuff" become major projects at large organizations. Every problem is a problem at scale.
I ended up owning a credit card merchant service processor when a customer hired my company, and then immediately not paying the bill - right when PCI was getting rolling. I'll never forget my first audit, where I was the CEO and lead developer for the MSP and the 20 something auditor was trying to disapprove my company because I had access to both production and development data. It took four days for them to figure out that because I owned the company, it was both acceptable, and unavoidable.
This is true for many many industries. McKinsey bills over $2k for small teams of 3 kids straight out of college. I was managing millions of dollars in advertising dollars while getting paid $12 as an intern.
Sometimes you pay for the resources, processes, and is intuitional knowledge put in place by a vendor rather than the warm body who is your point of contact.
Did this job as my first job out of college. Can confirm, I was considered “the techie one” because I knew the difference between a client and a server.
I quit after a few years to go into software engineering. OP, your first career choice out of college really doesn’t have to stick if you don’t like it. I have strong opinions about the audit/consulting industry but otoh it wasn’t a complete loss: as OP said, the soft skills from that time have proved genuinely invaluable.
I would distinguish between consulting and audit.
Nobody thinks most auditors are experts. I have met a few, but mostly their job is to perform very standardised assessments, to meet compliance requirements. Internal audit is no different in this respect.
Consulting, on the other hand, I agree is hugely oversold. I have experience of being invaded by a swarm of big-name consulting firm graduate "security experts".
They didn't do a bad job in the end. They gathered the requirements, listened to the staff, and recommended pretty much what we would have recommended. Someone somewhere with more experience must have reviewed it at some point I guess.
In this case, the senior management was really paying for the "big firm" seal of approval on a big spend, not for any earth shattering security insight.
i was always under the impression that when consulting for a company with a competent developer team, the best a consultant can do is to confirm what the internal team already recommended.
if the company is doing a risky project, having a second opinion may be well worth the money.
Programming consulting is no better. The crowning moment was being told during a 1-on-1 with my consulting company manager that I was being "too efficient" by automating and documenting everything I was doing.
Yes, a software agency makes less money if the people they employ are actually good at the job.
unless they are doing fixed-price projects. hourly billing is nuts.
Yes, and . . .
Yes, it's pretty scammy. Yes, if you know what you're doing, you know what they'll find. And charge you for what you already know.
But it's a tool. As a principal/staff-level IC or as a low-level manager, you may run into problems convincing the higher-ups to address very real security concerns. So you may want to recommend such an audit. Because those newcomers are "experts" and their findings may sway upper management. Even if management pays you to be an expert, they're paying extra for new, extra expertise!
As some one with 10+ years in info security, how do I get to be this actual expert In the room? Does it pay well enough? If I have to sell my soul to survive anyway, making more for it sounds good.
In my experience the actual expert does not need to be in the room. The actual expert wrote the standards that the college kids are auditing against. And even then, he or she is an expert in writing standards, which is not necessarily quite the same kind of expert you're thinking of.
You're way too expensive for this sausage machine.
This isn't specific to your security field, it's all over the B2B software space. Everyone's an expert on everything, and they're selling based on their branding or how good their sales people really are. It also helps that the people they're selling to are even less prepared and interested in the actual work, so it's all about how good you can make yourself look.
Its not the age of the person, its the experience level. The PCI internal compliance people (who were consultants) had never configured a server, never did sysadmin job of any kind, maybe they programmed something in college for a project 25+ years ago. All of the InfoSec audit/compliance people that I ran into gained their knowledge from books, not from experience. So when you challenged a "requirement" with an alternative or why in that system it didn't apply they had nothing to go on because they didn't understand the technology or how it worked.
Anyone with experience doing sys admin work will be doing that - or if they are in the infosec space they are probably doing pen testing, etc. using their skills and not consulting/compliance work.
Competence isn't the priority, the rationale for management to hire consultants is it moves the responsibility elsewhere to a firm that is blame-able and fire-able.
Consultants pretend to audit and companies pretend they're being audited.
The company is happy to say "we did it, we're secure! Nothing to worry about!"
The consulting firm is happy to cash checks.
One clue that a company is more interested in pretend work than real work is they start hiring consultants. No offense to consultants intended by the way. Most of the economy is pretend work.
I think the main issue is in perceptions of what the service is.
Maybe some think this person should be some industry expert, computer whiz, hacking genius who will guarantee an impenetrable system.
In reality it is as you describe - checklists of questions and procedures by someone who knows the basic concepts with managers who don't know the technical details. Nothing will guarantee an impenetrable system, so corporate is happy just to check the box that a review was done. They just want to make sure they aren't leaving the front door open so the insurance they buy will pay out.
I'm an application security champion who works on our teams systems to ensure they meet the company security standards. I have worked closely with some external consultants on a few reviews of systems.
I am close friends with someone who has been an IT/Cybersecurity Auditor for several years now and is apparently doing well (receiving praise/bonuses from upper management recently), despite on multiple occasions demonstrating they lack basic technical knowledge and don't understand what they are auditing. Apparently they just have a list of questions they ask and then fill out a spreadsheet despite having no technical understanding of what they are asking; e.g. on multiple occasions this person has said to me that they don't understand the difference between a server and a database. It boggles my mind that someone can be responsible for auditing a company when they don't understand simple concepts like servers or databases, but who am I to judge?
The usefulness of an audit is heavily influenced by whether the process is cooperative or adversarial.
And yet highly logical people on HN will simultaneously argue that the government and its bureaucracy is a great thing and we should increase regulation and rules in all facets of the tech industry.
The security side is literally the only part of tech that is remotely “regulated”. Self-policing organizations created their own standards and private firms conduct the audits. Yet it’s obviously bullshit and box ticking.
Do you not realize this is how everything in society works? It’s endless make-work where highly credentialed know-nothings check boxes in order to fulfill some bureaucratic process they didn’t invent and don’t care / aren’t capable of improving.
This is the end result of bureaucracy.
Some regulations are good, some are bad. Some just have purposes different from what people assume. Saying its impossible to regulate anything is just as stupid as saying its possible to regulate everything.
Yet there is an extreme aversion to “de-regulation” as if all regulation is good and any attempt at reforming bad regulation is reactionary and evil. The regulatory state is ever-expanding and contracts only rarely in very isolated instances.
3 replies →
As someone who is on the side interacting with auditors, it's pretty obvious they don't know what they're talking about. I've filled out numerous questionnaires where the questions themselves don't make sense, however, the auditors usually don't have the technical knowledge.
The whole process is frustrating because these certifications are supposed to improve security and usually they create busy-work that takes resources away from developing/implementing security.
Information security and cyber security are real fields. That there are unserious actors out there I don't doubt one minute.
I changed to infosec because of the money.
Next time go to software engineering if money is a priority but you still seek the intellectual challenge.
Cybersec pays well earlier than software engineering does, but you meet glass ceilings quite rapidly. Once you get really good at what you do, software engineering is where the real reward is.
Alternatively, if you don't mind focusing on a single product, becoming an expert in any commercial software that costs at least 1m$ to integrate in large companies will land you a high paying salary guaranteed with a tenth of the effort you'd need for software engineering or cybersecurity.
This is an interesting comment.
> Once you get really good at what you do, software engineering is where the real reward is
Do you mean any technological stack here.
> becoming an expert in any commercial software that costs at least 1m$ to integrate
Any particular examples here?
Thanks.
That’s interesting. From what I have seen, SWE is about 30% higher salaries on average. What field did you work in prior?
Well depends, how indepth the audits are being done, i mean not just simple port checks or MFA, but much more into server configuration, policies, permissions, cloud audits etc.
It's a self-serving industry. NIST, ISO, etc standards are also made to be self-serving.
I think the primary function of consulting is to provide an expensive scapegoat, no?