Comment by rckoepke

> SOMEONE

That's not necessary unless SOMEONE includes computer programs.

Yes, when things go very seriously wrong, I believe AWS can have literal people override that permission, which will leave a mile long audit trail and likely accompanied by an internet scale outage.

Except they get audited by 3rd parties on statements like that, and have controls tested. It's not like they're just ... digital ocean or somebody.

Yeah. True. I guess what I meant is that just a handful of employees have access to that and they need to have legitimate reasons.

Also, it is possible to build systems such that, no, there isn't a 'root' or 'unlimited permission' or whatever. Or that there is, but it's a multi-person credential.

This is one area where AWS takes things MUCH more seriously than it's competition, and they don't talk about it enough publicly.

The critical factor here is whether there are controls in place to prevent it. Sure, somebody probably could, but what to what lengths must that person go to do it, and what happens when it is discovered? Most things are not technically impossible, after all.

for its faults aws takes data privacy super serious. if you are in support you cant even see attachments customers put on cases without providing auditable justification

and you def cant see in s3 buckets or instances. hell if a customer sends you a link to an object in their s3 youre not supposed to open it

> Ideally customer is notified each time it happens - that would be cool, but likely technically not possible

You're underestimating the investments that AWS (and Amazon at large) make in to security, confidentiality, and auditing. You're also missing a fundamental implication of building AWS on AWS primitives.

As a relevant example there is only one AWS IAM and one CloudTrail. It's a core tenant of AWS IAM to put that control and root of trust in to the customers control. That means when developer support is helping with your ticket they do so via your accounts AWSServiceRoleForSupport role. That means you can control whether that role exists, which principals can assume it, the capabilities it has, and you can see those same API calls in your CloudTrail logs. Although it would make support difficult you're welcome to delete that service linked role and prevent support.amazonaws.com from assuming said role in your account.

https://docs.aws.amazon.com/awssupport/latest/user/using-ser...

This really isn't how modern security works at most cloud companies. Even if you have root class credentials or the ability to escalate to them in some way (and that's a big if by itself), its a LOT of steps to get access to customer data, almost always involving broken glass, many protection layers, and often requires cooperation of multiple other root level people/credentials from completely different teams.

Depending on how the infrastructure is built, or what the particular service set up, it may not even be possible to gain access to specific data without extraordinary means, possibly involving replacing physical hardware.

I already corrected my statement in another reply. You're right. I said probably only a handful of people can access customer data to do their job. I personally never met one. The goal of my comment was to illustrate that in my experience handling customer data there was a big deal. It's not like something you can casually query to see if a particular customer has a good business or not.

It’s the thing they tell you the most when you work there. Like in a a obnoxious way. Most infosec training is about that.

If someone has access to customer’s data for their work they have to do a bunch of extra training and do other stuff. Potentially sign some things and there’s probably a different way to authenticate. I really don’t know because I never had to do that and nobody I knew had that type of access but I heard when you do you have to put with more things.

As I said below this is something that they will talk a about like every freaking day. They talk about customer’s data as the most important thing to take care of.

Basically is preferable to get a bullet in the head than to ever reveal or tamper with customer’s data.

I cannot answer your question about who has access or not but I’m telling you what’s the culture when it comes to customer’s data.

At the end of the day I was just another IC doing menial work so probably not a good reference, but that was my experience

My reading of this is that the ex-employee used the knowledge about EC2 instance credentials being accessible as a path to gain unauthorized access to data. In theory anyone could have exploited this vulnerability even if they had never worked for Amazon. They never say that Amazon employees had privileged credentaials that would give them unauthorized access to customer data.

AWS customers that want to avoid this vulnerability should disable IMDSv1 as per https://aws.amazon.com/blogs/security/defense-in-depth-open-...

That leak didn't involve any insider access. So it doesn't prove that employees get access to the S3 data.

I used to work for AWS and had to deep dive into IAM to build a feature.

Basically Everytime you touch AWS your session is tagged with your credentials and has a unique ID. So everything downstream you touch has your session ID associated with it.

Now say somebody from Redshift wants to access the customer's data. They will then need to access to the encryption key in KMS. The trail will be there since KMS lives in the customer's account (you can audit your own access). And for production services, human actors cannot access these keys - only production credentials can. An engineer who can log into a prod host in theory can grab the temporary credentials there but it expires in 15 minutes so your trail will be rather visible. Also access to prod host has a high bar - only senior people can do it.

Now in theory somebody can coordinate with a malicious user in KMS team - but the bar is high. Also the actual master key never leaves the premise for KMS so your attack surface is very limited.

Of course there are some core teams like IAM and KMS where if they become vulnerable the whole thing falls apart. But that's a big stretch for those systems since they are the core to the business.

I can tell you generally how this works in Azure, I can't speak for AWS, but unless a customer is using BYOK for encryption of their data, I can't imagine how AWS c o u l d n ' t be capable of accessing data, and even then I wouldn't gurantee they couldn't still get your data. In Azure (as of a couple years ago), in order to access a customer's tenant it required VP approval, the support engineer was granted access for a specific amount of time, and typically only to specific services, all with the customers knowledge beforehand. It may have changed since the last time I had to go through this process and was restricted to blue badge employees. I have worked support cases since then and the support engineer would not even do a log me in/WebEx, etc session as they said they were not allowed to see the portal. But it may have been that they were not a blue badge and/or bcuz the customer was a critical infrastructure customer.

In order for AWS to comply with LEO's they must have some way of accessing data, that is NOT to say they do this for business purposes.

Other than the data each service actually retains themselves (i.e. the Lambda service themselves store your Lambda Functions because they need to execute them) customer data is generally stored encrypted at rest with KMS keys belonging to the customer (or sometimes managed by the storage team). It wouldn't be possible to peer into unencrypted data without persuading the KMS API to authenticate your access to the key. Presumably this capability exists, because otherwise Amazon wouldn't be able to honor warrants for customer data, but the premise that KMS is handing out decryption tokens for customer data for the benefit of Amazon Retail's business analysts is pretty silly.

And of course, you're always vulnerable to someone with access to the physical host of an EC2 instance where your workload is running. Only GCP AFAIK offers an encrypted-in-processing compute service, and it's like a week old.

https://cloud.google.com/blog/products/identity-security/int...

no