Security audits are just theater. If they were not, you could not ever convince them that using a platform feeding unlicensed source (including apparently from private repositories) to their commercial LLM is ever a pass.
Absolute theater. They do nothing to validate that you are compliant with whatever ISO cert you're pursuing. They make you install a root cert on your macbook and they say that's good enough to ensure compliance. You just attest that you don't do stupid shit like committing directly to master or testing in production and they believe you
It really depends on you auditor, audit approach and goals.
There are many audit companies which have a "under the hand" reputation of not properly looking and being easy to convince that you are secure, naturally at a above average audit cost (same but worse btw. for certificates showing compatibility with industry standards).
So if the audit was paid for by the company themself you can't trust it at all (which doesn't mean the company wanted to hide anything, this "bad" audit companies also tend finish the audit fast. So sometimes companies go for it, even if they don't have anything to hide).
Similar sometimes audit companies ask if they can audit you, this is for boosting their publicity using your name. This can easily turn into a "one hand washes the other" situation where they won't overlook massive issues, but still judge issues leniently.
Lastly there are some automated partial audit services which scan you public APIs/websites etc. Realistically they tend to be kinda dump, and might tell you they find a medium issue because (no joke) your REST API allows PUT and DELETE (1). Still I now take them a bit more serious after they pointed out, that there was a configuration error of a web gateway leading to some missing security headers.
(1: There is some history behind that, it's still dump for 90% of REST APIs)
Anyway, the situations so far are security audits which are at least 50% theater. BUT if a huge customers fully pays a audit company with a good/strict reputation then it often really isn't a security theater and can be quite a bad surprise if you company isn't prepared (because you have to fix so much). Like such reviews tend to not only be focused at your deployment or code but the whole software live cycle, including fun questions like "what measurements have you taken in case one of your developers tries to inject a supply chain attack" (which to be clear don't need to have perfect answers, just good enough, and most importantly clear and well documented).
we are EU based and have besides other attorney customers.
Cloud Act and more then just one or two cases of the US engaging in industry espionage against their allies(1) makes it a high legal liability to use more or less any service from a US company even if it's in the EU and a EU daughter company
On GitHub we only have some code, which always anyway goes through additional testing and analysis before hitting production, this is why it's barely okay. No code from GitHub directly goes to production.
The only reason we ever where on GitHub is because we didn't always had sensitive customers and switching CI over is always a pain.
So I don't know if imply them being incompetent for allowing GitHub or for wanting to not allow it, but both point have very good reasons.
(1): And I mean cases before Trump, the US (as in top government, not people) was always a highly egoistic, egocentric ally which never hesitated to screw over their allays when it came to economical benefits. The main difference is that in the past the US cared (quite a bit) about upholding a image of "traditional" values like honesty, integrity and reliability. Especially when it would affect their trade routes.
you could barely convince your auditors that using github was okay? well, my opinion of security audits is reaffirmed
Security audits are just theater. If they were not, you could not ever convince them that using a platform feeding unlicensed source (including apparently from private repositories) to their commercial LLM is ever a pass.
Absolute theater. They do nothing to validate that you are compliant with whatever ISO cert you're pursuing. They make you install a root cert on your macbook and they say that's good enough to ensure compliance. You just attest that you don't do stupid shit like committing directly to master or testing in production and they believe you
3 replies →
> Security audits are just theater.
It really depends on you auditor, audit approach and goals.
There are many audit companies which have a "under the hand" reputation of not properly looking and being easy to convince that you are secure, naturally at a above average audit cost (same but worse btw. for certificates showing compatibility with industry standards).
So if the audit was paid for by the company themself you can't trust it at all (which doesn't mean the company wanted to hide anything, this "bad" audit companies also tend finish the audit fast. So sometimes companies go for it, even if they don't have anything to hide).
Similar sometimes audit companies ask if they can audit you, this is for boosting their publicity using your name. This can easily turn into a "one hand washes the other" situation where they won't overlook massive issues, but still judge issues leniently.
Lastly there are some automated partial audit services which scan you public APIs/websites etc. Realistically they tend to be kinda dump, and might tell you they find a medium issue because (no joke) your REST API allows PUT and DELETE (1). Still I now take them a bit more serious after they pointed out, that there was a configuration error of a web gateway leading to some missing security headers.
(1: There is some history behind that, it's still dump for 90% of REST APIs)
Anyway, the situations so far are security audits which are at least 50% theater. BUT if a huge customers fully pays a audit company with a good/strict reputation then it often really isn't a security theater and can be quite a bad surprise if you company isn't prepared (because you have to fix so much). Like such reviews tend to not only be focused at your deployment or code but the whole software live cycle, including fun questions like "what measurements have you taken in case one of your developers tries to inject a supply chain attack" (which to be clear don't need to have perfect answers, just good enough, and most importantly clear and well documented).
From a company with a long history of leaking private data... That AFAIK never even claimed to have fixed their side of the Solar Winds issue...
from private repos? they explicitly say they do not
https://www.copilot.live/blog/does-github-copilot-use-your-c...
4 replies →
we are EU based and have besides other attorney customers.
Cloud Act and more then just one or two cases of the US engaging in industry espionage against their allies(1) makes it a high legal liability to use more or less any service from a US company even if it's in the EU and a EU daughter company
On GitHub we only have some code, which always anyway goes through additional testing and analysis before hitting production, this is why it's barely okay. No code from GitHub directly goes to production.
The only reason we ever where on GitHub is because we didn't always had sensitive customers and switching CI over is always a pain.
So I don't know if imply them being incompetent for allowing GitHub or for wanting to not allow it, but both point have very good reasons.
(1): And I mean cases before Trump, the US (as in top government, not people) was always a highly egoistic, egocentric ally which never hesitated to screw over their allays when it came to economical benefits. The main difference is that in the past the US cared (quite a bit) about upholding a image of "traditional" values like honesty, integrity and reliability. Especially when it would affect their trade routes.