← Back to context

Comment by anileated

2 days ago

Security audits are just theater. If they were not, you could not ever convince them that using a platform feeding unlicensed source (including apparently from private repositories) to their commercial LLM is ever a pass.

11 comments

anileated

Reply
shortrounddev2  2 days ago

Absolute theater. They do nothing to validate that you are compliant with whatever ISO cert you're pursuing. They make you install a root cert on your macbook and they say that's good enough to ensure compliance. You just attest that you don't do stupid shit like committing directly to master or testing in production and they believe you

  • dathinab  2 days ago

    > compliant with whatever ISO cert you're pursuing

    ISO cert compatibility audits are very different from a proper security audit.

    And weather they do anything to check if depends on which you high, many of the slightly more expensive ones have the reputation to be "fast" and "overlook most issues".

    But that doesn't apply to all security audits (but most audits for ISO compatibility, like really it's bad).

    Anyway see my way to long answer about the on a sibling comment.

    • shortrounddev2  2 days ago

      I'm certain there are good firms out there which will actually give you a legit audit and make recommendations. But if the client is not actually interested in security, there will always be unscrupulous firms who will essentially sell you an ISO cert for no effort required. In my experience, most medium to small sized companies place little value in security

dathinab  2 days ago

> Security audits are just theater.

It really depends on you auditor, audit approach and goals.

There are many audit companies which have a "under the hand" reputation of not properly looking and being easy to convince that you are secure, naturally at a above average audit cost (same but worse btw. for certificates showing compatibility with industry standards).

So if the audit was paid for by the company themself you can't trust it at all (which doesn't mean the company wanted to hide anything, this "bad" audit companies also tend finish the audit fast. So sometimes companies go for it, even if they don't have anything to hide).

Similar sometimes audit companies ask if they can audit you, this is for boosting their publicity using your name. This can easily turn into a "one hand washes the other" situation where they won't overlook massive issues, but still judge issues leniently.

Lastly there are some automated partial audit services which scan you public APIs/websites etc. Realistically they tend to be kinda dump, and might tell you they find a medium issue because (no joke) your REST API allows PUT and DELETE (1). Still I now take them a bit more serious after they pointed out, that there was a configuration error of a web gateway leading to some missing security headers.

(1: There is some history behind that, it's still dump for 90% of REST APIs)

Anyway, the situations so far are security audits which are at least 50% theater. BUT if a huge customers fully pays a audit company with a good/strict reputation then it often really isn't a security theater and can be quite a bad surprise if you company isn't prepared (because you have to fix so much). Like such reviews tend to not only be focused at your deployment or code but the whole software live cycle, including fun questions like "what measurements have you taken in case one of your developers tries to inject a supply chain attack" (which to be clear don't need to have perfect answers, just good enough, and most importantly clear and well documented).

marcosdumay  2 days ago

From a company with a long history of leaking private data... That AFAIK never even claimed to have fixed their side of the Solar Winds issue...