Comment by former-aws
6 years ago
Cannot up vote this enough. During my time both at Retail and AWS it was perfectly normal to trawl production customer data and come up with ideas to launch competing products. Prices were always set lower or free offering justified as data-driven and customer obsession. I hated the gas lighting their customers and left in disgust of the company and its leadership which encourages that behavior.
I know it's hard to do when you're making good money and would be going against co-workers.
But, if you see something, say something. This crap continues because there are too many folks that are happy to help support immoral business practices for some extra scratch. This isn't all on you in particular but when google folks started raising hell about Chinese censorship the company was forced to move. We all have the power to withdraw consent over how our labour will be used and, as software developers, we've got a strong enough employment market that we have real power to help make companies behave better - power that folks working in the warehouse are absolutely deprived of.
I mean the problem is corruption begets corruption. They WANT do to these things because you're going to get a massive bonus when the product you 'invented' does well because you stole the idea from an Amazon customer.
Amazon needs to be properly taxed so that this crap doesn't happen anymore.
The idea that they shouldn't pay taxes simply because they're large should absolutely enrage everyone.
This topic has nothing to do with taxes. They will always be trying to increase their bottom line whether that line is before or after taxes makes no difference. What is needed is a whistleblower. Not just a “when I worked for Amazon we did bad stuff”. We need that person to contact startup X, whose software and customer list was compromised. And then, this is key, share knowledge and proof of these accusations. Hell, do so through an attorney where you negotiate x% of resulting litigation proceeds of you’re worried about your privacy and financial situation. I’m pretty sure this would play out badly for Amazon in court.
1 reply →
The current zeitgiest is that taxes are Unamerican and tax evasion is American. Until that is fixed proposing solving problems with taxes is a pretty empty approach since people are happy to elect tax evaders to the highest office in the country and joyfully utilize services that are offered by companies that are famous for their tax evasion (Apple, Amazon, everyone honestly).
I think taxes aren't really a solution anyways - fines might be but taxes would hurt honest players just as much as dishonest ones. What they did is (AFAIK) illegal and needs to be punished, if it isn't then there is no incentive for them to correct their action.
Who is saying they shouldn't be taxed because they are large? There's no 'large company' tax break.
https://www.cnbc.com/2019/04/03/why-amazon-paid-no-federal-i...
There's specific credits/exemptions in the tax code that they are able to exploit (and perhaps they can only exploit some of them _because_ they are a big company), but it really isn't about their size.
5 replies →
What types of AWS data would be trawled? Are we talking about data inside S3 buckets, database schemas, particular architecure styles, the fact that a product is consuming {x, y, z} amounts of cloud resources, or simply "spending $m / year" in gross?
I worked in an area where it is really hard to figure out exactly what workloads were being run and where it would have been extremely useful to know even basic things like CPU utilization patterns, network throughput patterns, etc for a specific customer.
We had access to absolutely none of that information. We flew blind, relying entirely on the fact that we gave our customers enough hand-holding support that they would willingly volunteer information about their workloads so we could help them optimize it/save money.
No one even attempted to get more detailed customer information AFAIK because it would have been extremely against company culture. That isn't Earning Trust or having Customer Obsession. The idea of reading data in someone's S3 bucket or inspecting what is happening inside of someone's EC2 instance in any way was unthinkable. Amazon is huge and imperfect, but from what I saw AWS takes data privacy extremely seriously.
I can confidently tell you that Amazon's employees cannot see customers data inside S3 buckets or EC2 instances. They are extremely serious about that stuff since they know that will erode their customer's confidence.
But there's probably other superficial business data that's helpful to evaluate that.
> I can confidently tell you that Amazon's employees cannot see customers data inside S3 buckets or EC2 instances.
From a technical standpoint, that statement is false.
Every employee might not have the credentials to, but for AWS to function as it does, SOMEONE inside the company has to have those credentials.
If you change 'cannot' to 'don't', well then we've just gotta take you at your word, which is where we started anyway.
25 replies →
This is incorrect, at least from a logical POV and why it's hard to trust what cloud vendors say. A statement like this is either naive (most likely) or actively attempting to mislead.
Technically, its absolutely possible. Most likely you'll just need a support ticket or bug, and then you can troll around as engineer.
Also, security teams also usually have access to stuff when things get interesting.
Better to say that access is strictly on a case by case basis and monitored thoroughly.
Ideally customer is notified each time it happens - that would be cool, but likely technically not possible since data ends up in so many systems (like logs, SIEM, telemetry, debug files, backups, data scientist desktops,....)
5 replies →
Amazon is a massive company. How can you know this with confidence? Are you in the C-Suite?
7 replies →
1. Did you work on a team at Amazon in the likes of what user throwaway_aws mentioned?
2. What measures that you know of is Amazon implementing to make sure no employees across all teams are having access to said resources?
1 reply →
I'm sorry but what you just said is patently false:
https://www.bloomberg.com/news/articles/2019-07-29/capital-o...
Quote:
Capital One Financial Corp. said data from about 100 million people in the U.S. was illegally accessed after prosecutors accused a Seattle woman identified by Amazon.com Inc. as one of its former cloud service employees of breaking into the bank’s server.
While the complaint doesn’t identify the cloud provider that stored the allegedly stolen data, the charging papers mention information stored in S3, a reference to Simple Storage Service, Amazon Web Services’ popular data storage software.
3 replies →
Can speak for AWS. Only the later. Basically the usage information for cloud resources. This constitutes the foundation for billing. BTW, this is be true for any cloud, any SAAS.
There is no way an employee can look into customer data. There's enough trail inside AWS to prove that without any doubt.
What are the measures being implemented to ensure that no employee can look into customer's data?
13 replies →
Given how granular AWS billing data is, I would expect the odds to be fairly good that it alone is sufficient to make a good analysis for which third-party offerings are compelling markets. Then AWS takes their execution advantage, along with things like the lower friction that arises from first-party integration with IAM and billing, as well as not having to pay retail for the cloud resources, and it becomes very difficult to retain a moat unless you have a paradigm or perspective that is both critical to succeeding and is also incompatible with AWS culture.
You’re correct. It’s disturbingly detailed as far as what it reveals about architecture.
aggregated api usage stats, api client headers is often enough to identify competitor products and their traction, and is non-sensitive, coupled with account id to customers.
Do you have to use AWS to sell on Amazon?
1 reply →
Considering that OP created this account today and that they're admitting to what would be a felony and against Amazon's own privacy policy, I doubt this statement is true.
Even if the customer had a misconfigured S3 bucket that was exposed to the public, it would still constitute as accessing customer data you're not meant to see.
As other users have provided insight on, everything you do as an Amazon employee basically leaves a trail with your employee ID, even if you had access to private information (which you wouldn't basically because it's locked behind several layers of security). Fireable and sueable offense which Amazon would definitely not allow, let alone endorse.
> everything you do as an Amazon employee basically leaves a trail with your employee ID
That might be true in retail, but it wasn't anywhere close to true in AWS. When I left most engineers still had SSH access to the production hosts (and a not-insignificant portion of operations relied on that fact).
Leaving aside the question of what SSH access looks like today versus whenever you left...
There are many easy mechanisms to audit and monitor SSH sessions. So... no?
1 reply →
Definitely not defending parent here, but in this day in age many people create burner accounts specifically to avoid tying any statements back to them. It’s pretty acceptable practice to create burner accounts on HN. That said, I agree, I doubt any of these claims are true.
This frankly doesn't match my experience and I have to say I find it unlikely.
Before going into our AWS production S3 buckets, looking at our databases for customer lists AWS seems to be pretty careful to get an OK.
Now we are being told that production customer data was normal to trawl? How in the HELL are they passing all their certs with all production data so wide open. I do customer managed keys - I mean, this is a HUGE backdoor.
Either Amazon is lying about AWS security (and has fooled a bunch of others) or routinely trawling AWS customer production workloads for data is a false statement.
My understanding is that Customer Managed CMK in KMS only means that the customer has control over the key operations - like rotation, key policies, IAM policies, etc. AWS still has actual control over the KMS system and full access to the HSM.
Even under this definition how in the HELL are they "routinely" trawling our production data secured by these keys. I mean, does not one think that is rediculous?
This isn't amazon billing data etc (obviously I expect they analyze that carefully given they bring in billions from billing). To ROUTINELY go through AWS customer production datasets is beyond all reason.
No. AWS has no access to your material, nor is there a code path where they could get it.
1 reply →
I would assume the comment you're replying to means things like resource usage patterns and costs to estimate a client's profits for example. Rather than reading actual data from S3 or a database.
As I said to throwaway -- if you are of the mind to share, i am here to listen. my email is dai.wakabayashi@nytimes.com
Come on NYTimes! You can do better than email.
Don't ask someone to admit to felonies over email. Tech employers have a LOT of power to investigate their employees' digital behavior.
How about this instead: https://www.nytimes.com/tips
I want to be careful here, as I respect that you worked at AWS (that is, most likely), while I never have, and don't know what goes inside the company.
But it would be helpful if you broke that down a little more than 'trawling customer data', because at the most innocuous, if they're just looking at what's publicly selling on Amazon, what goes into sales rank, that seems acceptable, to me anyway.
I think there's a difference there, though. Retail sales and reselling are parts of what most people broadly consider the "same industry". I mean, a small seller making a deal with Amazon to resell something that they know Amazon could sell on its own is at least always aware of the competition.
In this case, tech investing and online retailing are not the same industry. Amazon is using a dominance in one to fund the other, which then it uses to either drive valuations of potential competitors down or to simply outcompete them.
And that's a plausible antitrust problem.
I'm normally not in the Amazon haters camp. Most of the time I'll defend them against the typical charges of unfair competition. Not this time. This is sketchy.
Stallman was right:
https://www.gnu.org/philosophy/who-does-that-server-really-s...
Hi former-aws: I'm one of the reporters and would like to hear more about your experience. Mind sending me an email at cara.lombardo@wsj.com so we can connect?
caralombardo: Please don't ask people to admit to felonies over email. That goes double for any FAANG employee; their employers have many options to surveil them. Your employer has a page listing better options
https://www.wsj.com/tips
In fact, I would add: do not trust a journalist that doesn't try to protect his/her source. Nothing personal, Cara Lombardo.
"perfectly normal to trawl production customer data"
It's not. And there are plenty of trainings inside of Amazon to make you aware of that. It is your fault, in the end, to not report your team. I have been on several teams at Amazon and this would always be an absolute no-go. It's already difficult to even get basic ideas about customer data, things that you would consider "essential" to improving the customer experience.
>> It is your fault, in the end, to not report your team
Talk about all time gaslighting. It's the managers/directors job to ensure compliance, not normal employees.
^^^ THIS
If you see another employee committing a crime, you're obligated to report it under US law. You can be considered an accessory if you don't.
4 replies →
you are wrong
Didn't you anonymously tip off the customer?