Comment by throwaway_aws
6 years ago
"An Amazon spokesman said the company doesn’t use confidential information that companies share with it to build competing products"
Maybe...but in the past, AWS proactively looked at traction of products hosted on its platform, built competing products, and then scraped & targeted customer list of those hosted products. In fact, I was on a team in AWS that did exactly that. Why wouldn't their investing arm do the same?
Cannot up vote this enough. During my time both at Retail and AWS it was perfectly normal to trawl production customer data and come up with ideas to launch competing products. Prices were always set lower or free offering justified as data-driven and customer obsession. I hated the gas lighting their customers and left in disgust of the company and its leadership which encourages that behavior.
I know it's hard to do when you're making good money and would be going against co-workers.
But, if you see something, say something. This crap continues because there are too many folks that are happy to help support immoral business practices for some extra scratch. This isn't all on you in particular but when google folks started raising hell about Chinese censorship the company was forced to move. We all have the power to withdraw consent over how our labour will be used and, as software developers, we've got a strong enough employment market that we have real power to help make companies behave better - power that folks working in the warehouse are absolutely deprived of.
I mean the problem is corruption begets corruption. They WANT do to these things because you're going to get a massive bonus when the product you 'invented' does well because you stole the idea from an Amazon customer.
Amazon needs to be properly taxed so that this crap doesn't happen anymore.
The idea that they shouldn't pay taxes simply because they're large should absolutely enrage everyone.
9 replies →
What types of AWS data would be trawled? Are we talking about data inside S3 buckets, database schemas, particular architecure styles, the fact that a product is consuming {x, y, z} amounts of cloud resources, or simply "spending $m / year" in gross?
I worked in an area where it is really hard to figure out exactly what workloads were being run and where it would have been extremely useful to know even basic things like CPU utilization patterns, network throughput patterns, etc for a specific customer.
We had access to absolutely none of that information. We flew blind, relying entirely on the fact that we gave our customers enough hand-holding support that they would willingly volunteer information about their workloads so we could help them optimize it/save money.
No one even attempted to get more detailed customer information AFAIK because it would have been extremely against company culture. That isn't Earning Trust or having Customer Obsession. The idea of reading data in someone's S3 bucket or inspecting what is happening inside of someone's EC2 instance in any way was unthinkable. Amazon is huge and imperfect, but from what I saw AWS takes data privacy extremely seriously.
I can confidently tell you that Amazon's employees cannot see customers data inside S3 buckets or EC2 instances. They are extremely serious about that stuff since they know that will erode their customer's confidence.
But there's probably other superficial business data that's helpful to evaluate that.
46 replies →
Can speak for AWS. Only the later. Basically the usage information for cloud resources. This constitutes the foundation for billing. BTW, this is be true for any cloud, any SAAS.
There is no way an employee can look into customer data. There's enough trail inside AWS to prove that without any doubt.
14 replies →
Given how granular AWS billing data is, I would expect the odds to be fairly good that it alone is sufficient to make a good analysis for which third-party offerings are compelling markets. Then AWS takes their execution advantage, along with things like the lower friction that arises from first-party integration with IAM and billing, as well as not having to pay retail for the cloud resources, and it becomes very difficult to retain a moat unless you have a paradigm or perspective that is both critical to succeeding and is also incompatible with AWS culture.
1 reply →
aggregated api usage stats, api client headers is often enough to identify competitor products and their traction, and is non-sensitive, coupled with account id to customers.
2 replies →
Considering that OP created this account today and that they're admitting to what would be a felony and against Amazon's own privacy policy, I doubt this statement is true.
Even if the customer had a misconfigured S3 bucket that was exposed to the public, it would still constitute as accessing customer data you're not meant to see.
As other users have provided insight on, everything you do as an Amazon employee basically leaves a trail with your employee ID, even if you had access to private information (which you wouldn't basically because it's locked behind several layers of security). Fireable and sueable offense which Amazon would definitely not allow, let alone endorse.
> everything you do as an Amazon employee basically leaves a trail with your employee ID
That might be true in retail, but it wasn't anywhere close to true in AWS. When I left most engineers still had SSH access to the production hosts (and a not-insignificant portion of operations relied on that fact).
2 replies →
Definitely not defending parent here, but in this day in age many people create burner accounts specifically to avoid tying any statements back to them. It’s pretty acceptable practice to create burner accounts on HN. That said, I agree, I doubt any of these claims are true.
This frankly doesn't match my experience and I have to say I find it unlikely.
Before going into our AWS production S3 buckets, looking at our databases for customer lists AWS seems to be pretty careful to get an OK.
Now we are being told that production customer data was normal to trawl? How in the HELL are they passing all their certs with all production data so wide open. I do customer managed keys - I mean, this is a HUGE backdoor.
Either Amazon is lying about AWS security (and has fooled a bunch of others) or routinely trawling AWS customer production workloads for data is a false statement.
My understanding is that Customer Managed CMK in KMS only means that the customer has control over the key operations - like rotation, key policies, IAM policies, etc. AWS still has actual control over the KMS system and full access to the HSM.
3 replies →
I would assume the comment you're replying to means things like resource usage patterns and costs to estimate a client's profits for example. Rather than reading actual data from S3 or a database.
As I said to throwaway -- if you are of the mind to share, i am here to listen. my email is dai.wakabayashi@nytimes.com
Come on NYTimes! You can do better than email.
Don't ask someone to admit to felonies over email. Tech employers have a LOT of power to investigate their employees' digital behavior.
How about this instead: https://www.nytimes.com/tips
I want to be careful here, as I respect that you worked at AWS (that is, most likely), while I never have, and don't know what goes inside the company.
But it would be helpful if you broke that down a little more than 'trawling customer data', because at the most innocuous, if they're just looking at what's publicly selling on Amazon, what goes into sales rank, that seems acceptable, to me anyway.
I think there's a difference there, though. Retail sales and reselling are parts of what most people broadly consider the "same industry". I mean, a small seller making a deal with Amazon to resell something that they know Amazon could sell on its own is at least always aware of the competition.
In this case, tech investing and online retailing are not the same industry. Amazon is using a dominance in one to fund the other, which then it uses to either drive valuations of potential competitors down or to simply outcompete them.
And that's a plausible antitrust problem.
I'm normally not in the Amazon haters camp. Most of the time I'll defend them against the typical charges of unfair competition. Not this time. This is sketchy.
Stallman was right:
https://www.gnu.org/philosophy/who-does-that-server-really-s...
Hi former-aws: I'm one of the reporters and would like to hear more about your experience. Mind sending me an email at cara.lombardo@wsj.com so we can connect?
caralombardo: Please don't ask people to admit to felonies over email. That goes double for any FAANG employee; their employers have many options to surveil them. Your employer has a page listing better options
https://www.wsj.com/tips
1 reply →
"perfectly normal to trawl production customer data"
It's not. And there are plenty of trainings inside of Amazon to make you aware of that. It is your fault, in the end, to not report your team. I have been on several teams at Amazon and this would always be an absolute no-go. It's already difficult to even get basic ideas about customer data, things that you would consider "essential" to improving the customer experience.
>> It is your fault, in the end, to not report your team
Talk about all time gaslighting. It's the managers/directors job to ensure compliance, not normal employees.
6 replies →
you are wrong
Didn't you anonymously tip off the customer?
As it happens, the Congresswoman who represents the part of Seattle that contains Amazon is on the House Judiciary committee, and may also very well be your member of Congress. Seems like something her office would probably want to know about if you could substantiate the claim.
https://www.wsha.org/policy-advocacy/legislative/u-s-congres...
(ignore the odd source of the link. it's the only place I could find her CoS and District Director's email addresses.)
It didn't sound like GP was saying their team did anything illegal – they accessed _public_ information about the companies they were copying.
It definitely feels scummy, but it didn't sound like GP had access to evidence of a crime. IANAL.
Neither "traction of products hosted on its platform" nor "customer list of those hosted products" are typically public information. They are information to which a trusted vendor might have access. There seems to be a fine line between trusting Amazon to sell and ship one's products and services without using its position to sell competing products and services, and trusting AWS to host one's confidential data without reading that data...
8 replies →
>It definitely feels scummy, but it didn't sound like GP had access to evidence of a crime.
Violating Anti-trust statues isn't criminal...but it is still illegal. Anti-trust violations also aren't the only potential laws this would violate. It sounds like it would violate unfair trade practices as well (most states has statues/laws/codes on point).
I'm not super familiar with antitrust laws, but it feels like they might apply here.
12 replies →
When I was at Google, we were encouraged by our lawyers not to worry about patents or unique parts of any product. If there ever will be a claim, they will drown the company in legal fees, so nobody is going to dare to sue us.
Patents were used, in many cases, as a form of research into a new area.
Not my google experience. They do say not worrying about patents, but that's because searching for patents could indeed make you liable as you were influenced by prior art.
Nobody at google even remotely mentioned "we will drown them in legal fees".
If anything, I have a huge respect for google legal.
Disclaimer: former googler.
Lots of anon accounts (reasonably) in this thread, so I want to back up as a non-anon former Googler that your experience matches mine. It wasn't "ignore patents", it was "don't look up patents so you aren't influenced by them".
10 replies →
My previous company was acquired by Google and I totally agree with assessment. Immense respect for Google's investing, corp dev and legal arms in as much as I interacted with them. They always treated us fairly and were ethical in their interactions.
> They do say not worrying about patents, but that's because searching for patents could indeed make you liable as you were influenced by prior art.
I've heard the same thing in startups and other companies. This is not something unique to Google.
That is actually patents working as intended.
Unfortunately the way patent law works now, make patents usually not work unless someone is ignoring the law.
Patents were created to give a reason for people to publish their "secret sauce" in a public manner, so anyone could read and copy them or create new products based on the patent.
If you DON'T want your product copied, the correct course of action instead is make it secret, for example this is what Coca-Cola does (they rarely, if ever, patent their products, and they hide the best they can their recipes and processes)
Those are then described as trade secrets and have their own protections as well.
1 reply →
Indeed
https://arstechnica.com/tech-policy/2018/06/inventor-says-go...
Amazon does it also with popular independent sellers. I know a motorcycle shop that was selling top quality products. Amazon representative contacted him if he would like to sell on Amazon, showed him offers how to stock and sell items. Amazon started a brand with the same name and their products were higher in search, were poor quality China made, jackets that were falling apart within a year. For the same price! Angry consumers targeted their anger to the real website leaving negative comments, not on Amazon! Shop owner had to change his brand after 18 years of being in business, as legal battle against Amazon would cost him more than the business had in stock.
This reminded me of the time Amazon started selling diapers at a loss to price-battle diapers.com. They won and ended up buying the parent company.
https://www.vox.com/2017/3/29/15112314/amazon-shutting-down-...
This is so wildly anti-competitive. I cant believe how many stories there are like this in this thread alone.
There's a House of Reps. hearing on Online Platforms and Market Power next Monday with Bezos attending. If anyone has some staffer friends, could be a good line of questioning to poke them about.
Please do this. The only way people like Bezos are held accountable are from people speaking up.
Not trying to come across as judgemental. But if I may ask, did you at the time feel like that was an ethical thing to do?
I joined after the team had gotten traction already. Both the GM and senior most product person on the team told me about their tactics independently.
To be honest, I didn't think of it as anything sinister at that time. AWS had such high octane culture to move fast and innovate that I actually felt what they had done was quite smart. It was a super competitive culture and people did whatever was needed to build new things. On a day to day basis the only pressure was to build... I don't remember instances where ethical guidelines were brought up. So, in a way, the outcomes were a result of what people were rewarded on.
Only after I left AWS I started thinking it was ethically iffy. I still believe Amazon is an amazing company and my time at AWS was one of the best learning experiences.
"It is difficult to get a man to understand something when his salary depends upon his not understanding it." - Upton Sinclair.
I wish we went into this in much more detail in high school when covering economics and ethics (if the school even bothers to teach ethics). It should be a prerequisite in any capitalistic economy (but not only those, it can easily be extended to other things).
I've also worked in industries that I think don't operate very ethically. It's amazing what you can ignore as an outlier because the alternative is uncomfortable or means you have to make a large personal change.
2 replies →
Interesting
Previous discussion of Amazon releasing a Basics version of an item at half the price:
https://news.ycombinator.com/item?id=11533973
> "An Amazon spokesman said the company doesn’t use confidential information that companies share with it to build competing products"
The above statement may be "true" if you redefine what is confidential. The Amazon MNDA in past years basically said that they could use any information they remembered from the meeting. I read non-disclosures carefully. I've never seen anything like it.
This is called a residuals clause, and it’s increasingly common. Be really careful looking for these - I won’t sign a vague/broad one, unless I am out of options. (e.g., acquisition or fail)
Ah, so that has a name? It was in the middle of the document in a fat paragraph. I was delighted to find it--kind of like picking up a big seashell on a crowded beach.
2 replies →
Just ... wow. This is an egregious abuse of monopoly power and is exactly the kind of thing that antitrust laws are supposed to address.
I was certainly naive when I heard about other big retailers who would refuse to allow any subcontractors to use AWS. "Surely Amazon has a Chinese wall" to prevent that kind of data sharing, I thought. Never underestimate the lack of morals in business is the right answer I guess.
> "Surely Amazon has a Chinese wall" to prevent that kind of data sharing, I thought. Never underestimate the lack of morals in business is the right answer I guess.
It’s remarkable to me how many competent programmers with years or decades experience in this industry don’t understand —- If you’re using AWS, Amazon has access to ALL of the data you put on AWS.
Not that they 'can' or 'want to', given the current state of technology they absolutely have to have access to all your data for AWS to function.
There isn’t currently a feasible technical way to work around this. And to head off all the ‘but FHE’ comments, see the ‘currently feasible’ above.
I'm not talking about not having any access in the technical sense. I'm talking about a "Chinese wall" whereby people who work for AWS supporting customers should absolutely not be able to inform any of the teams that build new Amazon services. These types of Chinese walls exist in many different industries, perhaps most famously finance, and when these walls have been "breached" in the past it has resulted in huge scandals.
1 reply →
I think enclaves are a more practical near-term solution for data privacy, but they don't prevent Amazon from identifying successful businesses based on e.g. resource usage growth.
1 reply →
Amazon does not access private S3/Ec3 data for retail competitive purposes.
The comments above indicating 'well someone has access' - yea, obviously, it's data hosting. Someone has access.
But the amount of conspiracy here is frustrating.
Amazon will play very aggressively within the bounds of the law, meaning, if they can glean public info about something, or look at their own sales data for a product, they will do that.
But to look at s3 data would risk the entire empire.
It's rational for people to be a bit skeptical, and so Walmart can say 'no data on AWS' but it's also an easy thing to do.
Now - is it possible that new retail PM, who used to be an AWS PM, and who for some reason still had access to things he shouldn't - went ahead and did that? That could happen. And maybe his boss finds out and looks the other way but calls IT and tries to have the loophole closed quietly. Etc.
As a policy are they trying to copy your product and even ask you for information and aggressively pursue customer data? Yes.
As a policy are they looking at your S3/ec2 data - no.
I dont think its just about playing by the book of law. I'm sure they also consider optics and trust in the brand.
1 reply →
[dead]
I was on a business call with someone from AWS on a different topic, and it was pretty darn clear they opened up some sort of Account page that discussed our (limited) AWS usage, and were trying to infer a bunch about our business from that. It doesn't even really matter how deep that data goes - even just month-over-month billing #'s or something like compute/bandwidth consumption is super telling.
We mostly only do CI type stuff there, so that didn't work so well for them, but if most of our revenue & operational use was through AWS, you bet I'd be worried about what they could infer.
This echoes the distrust felt by many Amazon Marketplace sellers that drives them to seek alternatives like Shopify.
Shopify is way way worse than Amazon. If you think Amazon is evil, Shopify is 10 steps ahead.
It's not just my experience. Talking to startups and warehouses in Canada, the stories are all about how Shopify invites for friendly talks and then stonewalls you once they have got the required information
Wow. I understand you do this as a throwaway but if true this is very bad stuff and it would be nice to have a lot more substantiation so that it could be verified.
I wish I could reveal team and product name... but that would be a career suicide. I'm not asking you to believe what I'm saying... but I truly am sharing my experience. I'd encourage you to talk to folks you know from AWS who were there for last 8-10 years.
Are you still with Amazon?
If you're building a competing product and using a monopoly in order to price gouge - nothing is throwaway.
The HN account is a throwaway, that's what they're talking about.
A clarification: I'm talking about tactics from 2010-2014. I left in 2015.
What I did not find from your post is in what manner are the data accessed. Is it at all publicly available? Is it metadata e.g. usage/billing? or is it production content like S3/lambda code/EC2 storage? It would be very helpful if you can clarify what kind of access it is.
Funny... I have software that combines well with something that has an online app store. They've been begging me to put my stuff in there. Nothing doing, I seen how you guys have embraced and extinguished others. They just took out their biggest app on the app store with their own version.
I’ve heard credible rumors of AWS teams using customer billing data to unpack what they’re doing in their accounts to inform competitive products.
> confidential information
aka, we did not sign an NDA with the party across the table
This doesn't matter if you don't have the millions set aside to defend yourself in court against a giant conglomerate. Giant companies breaking NDAs with scrappy startups is a story I have heard often.
This generally is not the case.
If you blatantly ignore and NDA, and then make a lot of money from it, then the 'small startup' will have a ton of money because the prize is huge, i.e. a % cut to lawyers who can work pro-bono.
Imagine you have a $10B company and some bonehead PM steals info from some small startup, for some stupid small project - it puts everything at risk.
In most case, I think you have boneheaded actors, usually not acting in the best interest of the company.
This is likely disingenuous. Large corporations like Amazon systematically refuse to sign NDAs with small players, hence none of the info is “confidential”. The rationale is that large companies might have people working on the idea already and they meet so many people/companies it would be restrictive for them to agree to any confidentiality.
It is a reasonable expectation that your data is kept confidential. If my hosting provider were to so much as look at my data without my explicit permission I'd sue.
Hi throwaway_aws: I'm one of the reporters and would like to hear more about your experience. Mind sending me an email at cara.lombardo@wsj.com so we can connect?
How is that not a form of corporate espionage?
Just a word of caution regarding the throwaways in this thread. Take them for what they are, anecdotal claims
A single comment from a throwaway account with no evidence but confirms my personal bias that Amazon is evil? I'm fully bought in before I catch myself.
The economic and reputation cost Amazon would take in ever accessing customer data to come up with some competing B-list product (say ElasticSearch as a managed service) is astronomical compared to potential profits. One thing I know about that company... they care about optimizing profit and are long term focused.
Please provide evidence for your extraordinary claim.
Whenever anyone asks for evidence I start to wonder why they need the proof. Why did you need this link? Do you have a business relationship with Amazon?
https://www.google.com/amp/s/www.wsj.com/amp/articles/amazon...
That's talking about Amazon Retail, not AWS
The customer data on Amazon Retail is Amazon's, not the seller's , just like the customer data when you buy shampoo from Walmart is Walmart's, not Procter&Gamble's
> Whenever anyone asks for evidence I start to wonder why they need the proof. Why did you need this link? Do you have a business relationship with Amazon?
I had made an oath to myself to not return to Hacker News and engage with the community after I was once temporally banned on Hacker News. But when I saw your stupid comment and recognized its disingenuous manipulation and misinformation, a pushing of an evil and harmful narrative ultimately made for whatever reason you might have, I returned to Hacker News to downvote your comment. Thanks for creating a vulnerability in the world for me to exploit the loophole in my self-imposed oath that allows me to return. I'm back.
1 reply →
[flagged]
2 replies →
Amazon and arguably google are in desperate need of a good anti-trust investigation.
Would Amazon be more, less, or equally liable if you used a cloud provider that then relied upon AWS for its hosting.
How would a startup that was concerned of Amazon copying it be certain to avoid such surveillance other than running its own data center?
if you are ever of the mind to share, i am of the mind to listen. my email is dai.wakabayashi@nytimes.com
Do you want to write an opinion piece on Shopify? I can get some minds to share
> AWS proactively looked at traction of products hosted on its platform
How is that confidential information if it's hosted on their own servers?
That is straight up some Darth Vader shit.
And that's why existing businesses joining the cloud pick azure.
What makes you think Microsoft would behave differently?
I presume it would be a lack of history doing the exact thing amazon are accused of doing here? That and the fact that they aren’t doing an e-commerce business in the way Amazon is.
Since Microsoft has always known to threat their business partners fair.
1 reply →
They are not as competent at sherlocking as Amazon.
1 reply →