OP runs a casino/gambling site. Gambling is a regulatory mess (I have spent far too long dealing with this as an RNG supplier), and so it's very hard to comply with every jurisdiction, and each one needs you to prove compliance to operate in that jurisdiction.* Gaming companies spend a lot on compliance and tracking, but since the internet is the internet, it's pretty hard to enforce perfectly, so some countries and ISPs take this into their own hands.
Due to that, IPs hosting gambling and gaming sites often get regionally blocked by internet providers or otherwise flagged as hosting illegal content. Those regional blocks consequently affect the reputation score of the IP, and if you are a traffic aggregator like Cloudflare, can cause other customers to have issues. One of the most aggressive and annoying regulatory environments for gambling companies is the US, so it's very possible Cloudflare has had some trouble due to gambling use of their IPs in states in the USA.
Cloudflare wanted them to use the BYOIP features of the enterprise plan, and did not want them on Cloudflare's IPs. The solution was to aggressively sell the Enterprise plan, and in a stunning failure of corporate communication, not tell the customer what the problem was at all. The message from Cloudflare should have been "Enterprise plan + BYOIP or ban, and maybe we'll work with you on price" but it was instead "you would really like the Enterprise plan."
*As an aside - we're lucky in that respect being a tech supplier with relatively uniform rules, but our customers (the gaming companies) get the short end of the stick here.
BYOIP is reasonable, though I doubt anyone actually does legislation blocks by IP. Since like half of companies on the internet use Cloudflare or other multi-tenant infrastructure everyone is aware that you can't block an IP address and hit one target. The only thing I've seen is DNS blocks (both DNS protocol directly and based on TLS SNI).
FYI, we also fully block users from the US (due to regulations).
My problem here is mainly the unprofessional communication and huge mess of mixing "compliance" with sales, without giving any clear information or options. And then the removal of our account without warning while we were still talking to them.
You would be surprised how big of a hammer ISPs will use when they are told to hit something. They live in a very different world than many modern web software companies - they are the plumbers for lots of things you take for granted, and look at the world the way a plumber does. Thanks to TLS, the plumbers can't see the HTTP headers to figure out what's actually flowing, so they sort of end up whacking all of it.
Generally, low-reputation IP addresses are associated with scams, spammers, and other similar things. Gaming somehow gets lumped into this bucket in some jurisdictions, but that hurts you worldwide (similar with other "sin businesses" like porn). These blacklists get published (I think there's some parts of BGP that make this happen, but I'm not quite sure what the mechanism is), and being on any one of them hurts your traffic everywhere because it becomes suspect.
I agree with you that this mix of compliance, engineering, and sales is gross. If this was the issue, they should have just told the OP.
While they absolutely shouldn't ban IP for reasons you said, some do that anyway.
The most (in)famous case is China's GFW which banes IPs all the time. Yes, other websites often get accidentally blocked, but they don't care. Moreover, you can't even communicate with them because there are no official legal regulations. This is something what any CDN or cloud providers have to deal all the time.
I also wonder if the company in the article didn’t know that (either by reading between the lines as you did or via other correspondence they didn’t mention) and weighed that in their decision to go with Fastly.
BYOIP isn’t just expensive—if your content is bad for IP reputation the time-to-flagging of your IPs is going to be way shorter on BYOIP than on shared IPs due to there being less dilution. And that’s without getting into the challenges around rotating/renting IPs on a continual basis.
I do agree that CF did not communicate that well or professionally—if the sales emails are the only comms that happened here.
I think it is possible that the company posting this didn't realize that this might be the issue, but you are right that they may know. It may have been a small company, even doing that much bandwidth. Online gambling sites tend to push an entire video game when you are playing on their site.
Many gambling companies are fine just doing BYOIP or running dedicated hosting infrastructure that is on providers who are explicitly running hosting for that industry (although they are moving to cloud). There is a good reason this separate infrastructure exists. In general, I would not assume they are rotating IPs: this is not a scam, it's a business, and they are largely fine with being blocked in places where they can't legally operate.
Maybe the first couple of weeks there was a misunderstanding, but by May 7 it was clear what the situation was. They was told they need BYOIP through and enterprise plan; they just didn't want to pay for it.
No, random.org isn't in that market at all, aside from doing some drawings local to them and unofficial games.
I think we are only one of ~3 TRNG suppliers who have been audited. Many games don't use a TRNG, though.
Since it uses atmospheric noise, you can also influence the numbers from random.org by transmitting radio waves in the area nearby - the operator of random.org has mentioned that there's so much RF activity that he is concerned about whether the bits are still random. A final issue is that they are also so low-volume that they probably can't get enough test data for the required audits (which can be a lot of data).
To underscore the volume question: Random.org used to have a running count of bits generated. The counter wasn't monotonic (before it broke in ~2015-2019), but the peak value I saw when I checked archive.org was about 250 GiB total since 1998 (that was in 2015). That is one quarter of the size of our "light" qualification test ("heavy" is 16 TiB). The RNG auditors also take O(100) megabytes for each audit, which would be a significant fraction of random.org's output.
I don't think it's a huge market, but state-run lotteries around the world need good random number generators for games without physical balls (like Keno for example).
I've talked with people that have created RNGs (rather than buying off-the-shelf solutions) and it sounds like soul-crushing work - mostly due to dealing with the government regulators that need to give final approval before the RNG can actually be put into production.
Considering they mentioned working with gambling/casinos, I would assume random number generator. Which may seem somewhat trivial to build a business around, except if you’re in a highly regulated industry like gambling that regulates the implementation of randomness (and probably requires auditing and other complicated things like that). I would love to read a blog post on all the complexities at here.
It almost sounds like you are excusing them. Asking people to switch to the enterprise plan and bring your own IP is reasonable, but not on a timeline of 24h, and trashing their account when they tell you they are talking to a competitor makes me feel like I should flee Cloudflare services with all haste.
This is just how people communicate now in the business world. It's up to you to read between the lines.
One thing I've learned to be wary of on the job is "do you need help?" That phrase is often code for "You are not performing up to our expectations. This is your first and only warning. Get in shape or get out."
The way Cloudflare approaches situations like this is not ideal for anyone.
You start using the service and don't pay a lot, so you make plans around a certain level of expenses. Then out of the blue you receive an "urgent" email from a sales representative and suddenly you have to go from $20 or $250 to $thousands right away.
Obviously it's not in CF's interest to keep a customer that doesn't pay enough, but dropping a "bomb" on the customer and make them feel like they're about to be kicked out from the service makes the customer lose trust on CF.
CF can probably match Fastly's price. If they had acted differently in this and other similar situations, they could keep the customer, be paid more, trust wouldn't be affected, and there would be no bad PR here.
Since the CF management that posts on HN usually say this is not supposed to happen, perhaps someone needs to sit down and look at the incentives sales reps have? Even if you don't care about the customers, this is affecting the CF brand a lot.
Why on earth any company would jump from $250 to $10k per month unless they had a gun to their heads? Even if their revenue is to the millions/billions (which most likely is considering the nature of their business). They work for their own profit, not Cloudflare's.
No one wants to pay more, but if they do it in a way that makes their customer run to a direct competitor and not want to come back, then maybe there's something wrong with their approach.
We only have one side of the story here, but it's not the first time I've seen posts/comments about these emails from Cloudflare and the messy communication that follows. As a business customer, I really hope I don't have to deal with any of this.
Seems pretty reasonable if the $250/mo plan is costing Cloudflare close to or more than that amount of money due to any loophole or other unforseen expense in the plan.
What seems interesting to me is just what the loophole is and how many other business are also on the radar for this drastic pricing change. Are there other goodwill discounts Cloudflare is ready to start collecting on, or does the gambling site represent a unique situation?
have you seen the twitter post from Gamdom about mentioning the fastly quote in negotiations? they were taken down instantly. It doesn't sound safe to mention competitors with them or even exercise the prospect of leaving
I will remind HNers: is Cloudflare not the company that leaked sensitive data through cache files that were indexed by at least Google, and when the tech community were up in arms about the massive leakage of sensitive data, the CEO’s strategy was to turn up here and criticise Google for not deindexing quickly enough?
That's one of the main reasons I'm leary about them. Such a big f-up is difficult to forget. It shows that they have a move fast and break things culture which for a company that is responsible for critical infrastructure feels wrong.
In response to this incident Cloudflare has made big engineering changes, including huge work to move away from C as much as possible.
The offending parsers were rewritten in Rust (https://github.com/cloudflare/lol-html), as well as WAF, image optimization, and a few others. Nginx is being replaced with a custom cache server.
New implementations are using either the Workers platform, or are written in Rust or Golang.
I interviewed there once and they asked me what I would do if a service broke after a deployment. I said the first step was to revert to the last known good version and then investigate. Color me surprised when that was not the answer they expected.
I remember them criticising Google for not being faster at removing cached files. I don't remember them blaming Google for their screw up.
And let's be honest, if a big provider wants to offer cached versions of pages, they probably should have a way to purge those files in case there's a problem (eg: malware).
This was a much needed reminder. Although, it's quite difficult to find a better DDoS mitigator which is better than CF, I still wouldn't trust them for everything. Especially, since they are most likely snooping on the decrypted HTTPS connections
> Although, it's quite difficult to find a better DDoS mitigator which is better than CF, I still wouldn't trust them for everything.
Adding Challenges, TLS fingerprinting and Rate Limiting is possible on just about every major CDN platform to be honest. I guess with CF it's more "ootb" though, where you don't really have to think too much about policies - but at the same time, you can't go as granular in those policies (e.g layered) as some others.
At the moment the account got banned, I would guess that the CloudFlare sales team had this down as a "60% likely to close, estimated close in 6 weeks".
There is just no reason they would suspect that they were going to lose the deal to Fastly at this moment. They were very much the default winner.
Extortion or not, I just can't fathom that they ragequit the deal at this moment, because they were about to win it.
It therefore seems likely that after looking into it they disqualified it as a business category which is against their TOS or whatever.
Or that the enforcement and sales teams have very similar, overlapping triggers for engagement, etc.
Cloudflare's behaviour here was shitty and this is not the only report. By all means their reputation is very generous free tier and a horrible experience in paying.
BUT seriously who ragequits a winning deal? Another comment summed this up - the attention caused them to take a look and realize they don't support shady-ish casinos, possibly (seeming to) evade US legislation, etc.
> It therefore seems likely that after looking into it they disqualified it as a business category which is against their TOS or whatever.
The first sales email is from a Cloudflare with “Gaming Division” in their email signature, so they were clearly aware of the nature of the customer’s business. Moreover, it seems they have an entire department dedicated to serving the gaming market.
It's quite a common pattern in saas. Someone gets through automated ToS checks with some niche use case and escalates some unrelated issue to support which triggers manual ToS review. That being said, a corporation as big as CF with 120k usd bills should do better and never let this happen. Very amateur.
Sounds like OP is a casino and plays domain games to avoid regulatory interest. Recommend reading article carefully before reacting to the headline. Hopefully Cloudflare provides a perspective.
Hmm. My take is the casino structured its business to comply, not to evade interest. Further, I don't see how Cloudflare benefits by taking on the risk to charge more to help a customer avoid scrutiny. More like: they know it's a humming business and want a piece.
The way I read the screenshots of the emails from the articles seemed to suggest that something the authors company was doing was causing issues with IP reputation on CloudFlares range.
Them very aggressively highlighting the BYO IP feature and then even suggesting third parties to rent IPs from strikes me as a significant detour from their normal “script” (having dealt with their AU sales team before).
> We do have multiple domains that mostly act as mirrors to our main domain. We have these for a few reasons. One is that since we are a casino, we have different regulatory requirements we need to comply with in many countries.
Evasion:
> Another is that we use them to target different global user groups and affiliates and track conversions long-term. This also means that if a country DNS-blocks our main domain, a secondary domain may still be available.
This is more like one gang hitting up another for "protection" payments. I had to laugh when they called it "Trust & Safety".
> My take is the casino structured its business to comply, not to evade interest
It's impossible to say what's going on since it's an anonymous post with no details.
Maybe it's all 100% true.
Maybe there are some key details being left out. Wouldn't be the first time I've seen one of those outrage posts that seriously misrepresented things.
Whatever the case, obviously the author is not an unbiased party. These posts do well because "zomg Cloudflare bad!", and maybe they are, but I sure as fuck don't trust some casino guy either.
A reasonable scenario to me seems to be: An automatic "upgrade to the enterprise plan" requirement was triggered, and then in the process of the sales calls to make that happen, Cloudflare got serious eyes on the customer for the first time (whereas at a paltry $250/month previously they wouldn't have), and realized exactly what line of business the customer was involved in, and decided to fire them.
If it's legal but burdensome (somehow) to host a particular industry, requiring more money to deal with the increased burden seems reasonable. For instance, if their legal department needs to deal with complaints from various countries, that probably costs more than $250/month.
That being said, I doubt that's the core issue in this case.
If you think something your client wants could explode into a liability, you can turn them away or you can just make sure their bill covers your exposure.
If it's a legally questionable service, there's likely to be plenty of abuse contact, or they're going to be a big target of crime, they're going to end up paying more. This is the same reason why some industries (eg porn sites) have always paid more for card processing.
It's not just 10k a month. it's 10k a month for the plan that allows you to BYOIP (Bring your own IP addresses). That was cloudflare's issue.
Their business was causing IP reputation damage and all plans but the enterprise BYOIP plans share the same IP pool.
Essentially it was "use your own IP pool and pay us for the cost of maintaining that pool for you or GTFO".
This wasn't just a normal sales rep hitting them up. This was trust and safety (i.e. the moderation team) coming to them with a compromise that would allow them to stay on the platform. They chose against that and were dragging their feet.
The timeline of the article also really makes this clear. This wasn't over the course of 24 hours. This started a full 4 weeks prior with sustained back and forth. They only included a few images of emails from the discussions but the article makes clear that there was more discussion happening.
And to quote the article. After receiving the ultimatum, they got an entire extra week to deliberate.
> We managed to buy a week of time by letting it escalate to our CEO and CTO and having them talk directly with Cloudflare.
Then finally when they told CF that they were just buying time while looking to move elsewhere, CF dropped their act of goodwill and the moderation team resumed the moderation action they would have taken in the first place had this been a smaller account.
----
So yeah it sounds bad from the snippets but this was basically "hey you are a big customer and you are breaking rules we would normally ban anyone else for but if you can compensate us we'll spend the labor hours and infra to let you keep operating in your own little quarantine box.". So this really should be seen as an act of goodwill rather than malice.
I can reason my way into it, I think objectively. To protect their IP reputation, CF required BYOIP. This costs them something, and de-jure requires an Enterprise plan. Which for the customers usage costs $X. Is it right? Ehhhhhh. Does it follow corporate logic? Yeah. (Sales logic? YES)
I'm not defending Cloudflare's exact actions in this scenario, but it seems reasonable that there are cases where yes, for $10k Cloudflare is okay.
Risk can be mitigated, especially if you take care to know what the risk is, but risk mitigation and the salaries of the risk mitigation teams are not free.
The answer of "no, we will not host you unless you pay us enough money to hire people to make sure we're not breaking laws by hosting you" makes plenty of sense, and an online casino that is likely dubiously legal in many countries is definitely a place where you might use that answer.
I'd also expect there are cases where Cloudflare enter into enterprise agreements with customers, get a good hard look at exactly what's happening, and then tear up the agreement and walk away.
That's not true at all. That line of argument gets close to "if this product is free for open source, why is it not free for me? either it costs something to operate or it doesn't." You don't get to price the service.
The point is more that the author is an unreliable narrator and you need to apply a little salt to the rest of the story. Cloudflare absolutely shouldn't be taking bribes to permit regulatory evasion. But if they are, I want more evidence than a substack post.
Taking a step back, why would they even care if their platform is supposedly neutral and not responsible for the content ?
If they can indeed stop providing services to a casino, why cannot they shutdown a website spreading pro-war propaganda, or a website selling illegal services ?
It means they are making editorial choices, instead of just being the technological provider and being a neutral "internet pipe".
Not sure it's really in their best interest to self-police in the end, as they could lose their DMCA safe harbor provision ?
> Taking a step back, why would they even care if their platform is supposedly neutral and not responsible for the content ?
Because their main network all uses one big IP address pool and the blocks by various regions/countries against their site were probably not just DNS blocks but also IP address blocks.
So they now have an account whose activity is getting their IPs banned in countries where they operate.
So they told the account owners they needed to pay for an enterprise account and a dedicated IP address pool maintained by cloudflare. That's why CF kept talking about BYOIP in the emails.
i.e. "Pay for us to build you a quarantine with your own IP pool or leave ASAP"
I do encourage you to read the whole article cause there is some fine details in there. The main point is that we were happy to remove any domains apart from our main domain (which gets > 95% of our traffic) but Cloudflare did not give us that option or any other detail on the supposed issue.
If 90 or 95% of their traffic comes from a single domain (and presumably has for a while), that still doesn’t make OP sound guilty. If there was a legal issue Cloudflare legal should’ve stepped in, not their sales team.
That was the part that bugged me. This workflow is very busted from a user standpoint, though I'm sure it works very nicely to Cloudflare!
It smells like the "problem" was detected by automation, but instead of being able to reach anyone technical to work through it, you can only call sales teams.
This shouldn't matter, in general Cloudflare responds to complaints about allowing illegal content with "we're a neutral utility, we forwarded your complaint to the site's webhost". To me, the article showed that Cloudflare was being extremely aggressive with selling the customer on an enterprise plan and repeatedly invented excuses to get them on the phone with their sales team. They then took the site offline and locked them out of their account when the customer started talking with other CDNs.
So the thing that stands out in the article, is that cloudflare's initial communications (and the final communications, when they moved to ban) implied issues with their behavior (trust and safety team, terms of service violations), but in between it sounds like the didn't talk about ToS at all, just sales team asking them to buy enterprise. Though it's possible OP is omitting some explanation given by as why enterprise plan would alleviate ToS issues.
Well HN is the unofficially official Cloudflare Support forum. I think we will hear from them soon. From past experience normally their response time for anything Cloudflare on HN is within 2-3 hours.
Except Cloudflare position here is not to ban them but they want to get paid for it. You are shaming the OP and his business but the reality is that Cloudflare has acted in a worse manner and that should be highlighted.
For $10k / mo paid 1 year in advance, your cloud provider does a legal review of the situation and figures out how to make your problem work on both the technical and legal level. It's not a "special plan", it's consulting.
Edit: "How do you know?" -- I don't know it's actually what happened, but when switching to enterprise, you don't go from 10% margin to 98% margin. The added costs actually represent added budget for the provider to deal with your "special case". ALL enterprise pricing tiers are disguised consulting contracts.
It's 10k a month for them to set up a dedicated IP address pool so that they could BYOIP and buy their own IP addresses instead of getting the IP addresses in cloudflare's main IP address pool repeatedly banned or reputation harmed.
i.e. it's a $10k fee for maintaining the infrastructure for a quarantine around their services
"Now this needs a bit of context on what they are talking about. We do have multiple domains that mostly act as mirrors to our main domain. We have these for a few reasons. One is that since we are a casino, we have different regulatory requirements we need to comply with in many countries. For example, many games are only available in some countries. Some countries we block completely. Then we have a few different domains that remove certain game groups or site features - for example our social features (chat, user tipping / interaction) or our sportsbook. Another is that we use them to target different global user groups and affiliates and track conversions long-term. This also means that if a country DNS-blocks our main domain, a secondary domain may still be available. This could arguably be seen as a violation of the Cloudflare TOS, as they wrote above."
Looks like they COMPLY with regulatory interest, to me.
When it comes to laws and taxes, "comply" and "evade" tend to be synonyms.
"In order to comply with tax regulations and donor laws, we had to structure our activities in order to make it possible for political donations to be classified as regular consulting income".
If that's the take, that means Cloudflare is okay with 'breaking laws' so long as they can take a heavy cut of the ill gotten gains? </sarcasm>
Let's not try to find reasons to harm the messenger and stick to the facts -- a paying customer was suddenly extorted for hundreds of thousand of dollars out of nowhere.
Using localized versions of your services to comply with regional laws and enhance user experiences (i.e. make money) is SOP for practically every international $bigco. Online gambling is regulated and legal in ~50%-70% of the world; without actual evidence to the contrary, it’s completely reasonable to assume that this is a legitimate business. I’m really struggling to agree with the “two sides to every story” replies being left here about how there’s likely shady activity going on behind the scenes, when to me the post read as candid and transparent about the nature of the nature of the business, the admitted legitimacy of CF’s TOS violation claims against them, and the content of the communications with CF.
My 2c: It’s scummy that CF did this. It looks like they were disingenuous about the severity of the violations and used it as an excuse to get more $$$ from an already paying customer to make the manufactured problem go away.
Nah, you have different domains so you can track and maintain flows, also the regulations might even stipulate having domains in the locale, the headline is very much accurate after reading the article.
I mean, sure, they’re probably doing some sketchy regulatory dodging or whatever. Which part of this can Cloudflare solve by having them pay $120k/year to them?
Over-charging is a legal way to effectively deny service. When I'm offered jobs I don't want, I sometimes tell them my salary is 3x what I really need.
Yeah, there's some pretty key info being left out. I don't doubt that Cloudflare communication sucks especially when dealing with their sales team (aka bizdev which is what OP was originally contacted by), but the second screenshot is pretty damning.
My guess: Their account fell out of the non-enterprise TOS for some reason which is being obscured in the post (probably domain rotation related). Their T&S team proposed moving to enterprise for a custom resolution. OP's company refused, their account was purged because they had gotten several warnings about it.
I'm sure this sounds frustrating to the average HN dev who runs a legitimate startup with cloudflare on top and is now biting their nails worried to death about what will happen to them. But "online casino" immediately raised a million alarm bells in the post.
I did mention the multiple-domains issue in the post. It would not have been amazing for us to remove our secondary domains, but we would have been very happy to do it if it had resolved the issue. We asked them again and again but they would not give us any detail or options apart from their 120k/year package. Note that BYOIP (which I guess they could reasonably have required to isolate us even if we only use a single domain) is available for a fraction of the cost elswhere (e.g. fastly).
Since we already left Cloudflare the only reason I finished writing this article is to warn others. I think it's still relevant to many companies regardless of what you think of casinos, since very unprofessional sales tactics (unprofessional as in business threatening) seem common place with them. Do look at the linked other posts and comments here from other people affected that don't have anything to do with casinos.
Is the casino illegal in the jurisdiction they're based out of?
It doesn't seem so, so there is at least a valid reason for Cloudflare to keep them as a customer as they're not violating the laws where they have their business in.
This has been my experience with 80+% of these loud complaints about services, especially regarding "losing Google traffic". Dig into it just a little and you find out the complainer was doing something extremely shady that the service is often too polite/proper to call out in a public forum.
Cloudflare was the company that went viral for the firing of an account rep not hitting her goals. I wonder if it’s overall indicative of not a great culture in terms of relationships with enterprise customers.
> Cloudflare was the company that went viral for the firing of an account rep not hitting her goals.
How is that something worth going viral over? Salespeople get fired all the time for not meeting their sales goals. Engineers similarly get fired all the time for not meeting their productivity goals. If you don't do your job well, don't expect to keep it.
And if I recall correctly, in this particular case, she was a green employee who hadn't even made a single sale yet! What more obvious of a layoff target is there than that? Would you keep a green unproven salesperson over a proven veteran salesperson who's landed 9 figures in sales?
While I agree with you, I think the call is for companies to be less psychopathic and stop onboarding people within 90 days of mass layoffs.
Especially in a world where people pick up their whole lives and relocate for jobs. Recent joiners aren't getting any sustainable kind of severance either. The idea is if you're hiring them you have a minimum commitment to support their success.
Yes she was an obvious fire, but it's also the organization's fault. Enterprise deals also take way longer to close than that...
All that said, salespeople can and do move jobs a lot. I'm sure she'll be fine.
She did say she was around for only 6 months and enterprise sales cycles can last 12+. Though I guess if you’re engaging in scammy behavior it can be much less… maybe she wasn’t willing to do that.
We had a site hosted on CF business plan with fairly large bandwidth usage (completely legal, had a lot of media). They approached us with an enterprise plan but we did not have the budget for it.
Asked for a little time, they said fine and we moved much of the bandwidth usage to a couple of dedicated servers on OVH I think.
Can any1 explain how HN algo works that this post, which at time of writing has 355p, 180comments while being posted 1 hour ago, isn't even on first page (ranked 31)???
It set off the flamewar detector, got flagged by users, and got downweighted by a mod.
The 'customer support of last resort' genre is common and not usually a good fit for HN [1]. If people feel this story is unusually relevant and interesting, I'm not sure I agree—long experience has taught us that one-sided articles like this nearly always leave out critical information—but I also don't mind yielding in an occasional specific case, so I've rolled back the penalties on this thread.
The issue from our point of view is not about story X or company Y—it's a systemic one: the most popular genres of submission (especially the rage-inducing ones) get massively over-represented by default, so countervailing mechanisms are needed [2] if we're to have a space for the more intellectually curious stories that the site is meant for.
I would be very happy to hear Cloudflare's actual side of this. (Or - it would have been great if they had given their side to us before getting into this mess). The only critical information from our side that I'm aware of is that we're a casino with multiple domains - which is why I put that right at the top. But most of the info should be relevant to any business interacting with CF.
I do admit that I originally drafted this article as a "customer support of last resort", since that seems to work well for CF specifically. But it's too late for that anyways by now - the problem is "resolved" by fire and we don't plan to move back.
I purely posted it now as a precautionary tale for other people because of all the pain it has caused us. So the audience is tech people in most companies of small size that will hit more traffic at some point in the future.
if you check sites that track HN's rankings, it was ranked #1 for a while, then it suddenly dropped to #27 and continued declining
https://hnrankings.info/40481808/
thanks for sharing. I didn't even know this was a thing! Comparing to 4 or 5 others, this definitely looks more like a step function into obscurity unlike the other lol
Yeah I commented, refreshed and was shocked to see it disappear. It's ironic that we're (reasonably) asking for transparency about a post which is kind of about non-transparency.
1-The gambling business is shady by design, whether you like it or not. This was probably more risk than benefit for them based on what they were getting from you.
2-Your business is probably very profitable, and $300 a month is very cheap compared to the potential hassles they could face working with such a business.
3-I find it very inappropriate to dox business representatives and show names when you have carefully hidden any information regarding yourself and haven't even disclosed your company name.
After all they can choose with whom they want to do business. They gauged what price they could ask you, factoring in how profitable your business is and how noisy and painful it might be to work with you. It sucks but this is the downside of SaaS/PaaS.
Gambling site or not: Cloudflare took their money for years, failed to communicate any problems, then deleted their data when they didn't accept their "enterprise deal". There's nothing saying that they won't do the same to ANY of their other thousands of customers, many of who reads this forum...
To (1) - if this was the case, it would have been great if they had talked about it openly or in any way really. To (2) - I do agree that $300 is probably cheap. But I also think that $10k is very expensive, and it seems Fastly agrees.
(3) Mh, I don't think this is doxxing and didn't expect having names would be a big problem. I've just updated the screenshots anyways and censored the names of the representatives.
Cloudflare of course chooses who they want to do business with, but they also pride themselves in being neutral.
Cloudflare certainly handled this poorly in their execution and abysmal level of transparency, but they’re almost certainly purging loss-leading risky customers like OP and they really don’t owe OP the time of day.
OP is lucky CloudFlare even gave them 24 hours. I’m not going to dig through the their TOS or anything but I’m going to guess that you need to have an Enterprise contract to be a business of certain categories like banks/crypto, pornography, and gambling, which explains why they were being connected with a sales team.
OP mentions lost customer trust…but Cloudflare doesn’t want or need OP to trust them. $250 a month isn’t enough to deal with a business like that.
OP isn't the only customer whose trust they lose by handling the issue in this way. It's fine if they want to terminate a relationship with an unprofitable or risky customer, but doing it with insufficient notice to make other arrangements is pretty extreme. In a case of blatant abuse, that might be reasonable, but that doesn't seem to be the case here the way OP tells it.
I did quickly search the TOS for the word "gambling" and did not find it.
I don't know how those situations work though. The customer is obviously allowed to say whatever they want, but if a Cloudflare employee or CEO disagreed, would they be allowed to provide their own version of the facts? Wouldn't that go against privacy rules in some way, showing the details of someone's account? I would think they would only open themselves up to legal trouble.
As far as I can see, the author was careful to redact their domain from all screenshots.
- "This also means that if a country DNS-blocks our main domain, a secondary domain may still be available. This could arguably be seen as a violation of the Cloudflare TOS, as they wrote above."
Attorneys love it when people put everything in writing like this.
If a country A decides to block twitter.com but forgets to ban x.com which remains available ... is Twitter engaging in illegality / violation of CDN terms of service?
Like most things in the legal system, it depends on intent. It's pretty obvious that twitter's rebrand to x.com was an actual rebrand and not some way of evading domain bans.
This has always been my concern about establishing a presence online. I've considered blogging about my experiences at work or the cool stuff that I've built and it feels impossible for me to know when I've crossed a legal boundary. How do I know for sure if I'm sharing proprietary stuff or confidential stuff. The lines of legality seem to get blurry real quick.
From personal experience I know that 10TB per month is like 30k/year and SSL for SaaS is around 40k/year on the enterprise plan.
No idea about pricing for having your own IP.
I have no idea why Cloudflare would ask you to use these two features. SSL for SaaS is only useful if you want to add domains and certificates via API.
I have had my fair share of negative experience with Cloudflare but this is next level bad. Unfortunately companies can chose who they want to do business with but it shouldn't be like this.
From personal experience I know that 10TB per month is like 30k/year
What the hell? That's way more than AWS costs, 90% of which would be egress fees. And cloudflare has done a lot of marketing to rightfully call out those egress fees as far too high.
It's the whole enterprise plan, you can't only buy traffic. So you also get all the features which you don't need or want as you can see from the screenshota shared in the original post.
Even on the enterprise plan they don't really start to talk to you about traffic until you are like 3x over your contracted traffic for a couple of months.
It sucks, it feels like they are competing against themselves because they don't have clear pricing or limits.
> Unfortunately companies can chose who they want to do business with but it shouldn't be like this.
If you have a contract with them then they can't arbitrarily choose who they do business with. OP would presumably have a chance at a lawsuit against cloudflare here, the success of which would depend on how well cloudflare argued the ToS violation. A lawsuit might not be worth pursuing here, but this isn't a case of "it can't be helped".
This is my first post on Hacker News as I primarily just browse. This situation kept me intrigued, wanting to know how it would unfold.
The Google Cloud situation and all these little happenings, including the proliferation of Gen AI into everything, make me long for the days when companies had their mainframes onsite, in closets or separate rooms, away from CDNs and cloud networks. It seems like a better idea to use these cloud networks as a separate off-site backup rather than for primary use.
I’d love to learn more about what will happen next in this saga. I’ve seen a post where a Cloudflare exec has posted here on HN before. They probably won’t say anything for legal reasons, but what repercussions can Cloudflare expect for this? Will they be, or can they be, sued for this downtime and the related expenses?
Unfortunately it's difficult and expensive to scale those traditional solutions to the modern world of billions of internet users located all over the world. It's still quite slow to access a server on the other side of the globe. It's less noticeable for an american user accessing an american company's resources.
Does it matter for 90% of business?
I will say no, the vast majority of business is rather localized for very good reasons (as a starter, language/culture/legalese) so it is an unnecessary "feature" most of the time.
The reality is that manager drank the Kool-Aid and wants to pretend that they are Google/Facebook or the likes but almost no one else has this kind of scale.
Outside of tech behemoth nobody need that cloud bullshit.
This doesn't pass the sniff test. From the actions Cloudflare took and their communication, it's very clear that there was something about the way their services were used that they were unhappy about. The post doesn't include what that problem was, but I have a very hard time believing that the author was not in the know and just got their account nuked without any further commentary, especially after being in a number of calls with real human beings from Cloudflare. Surely they'd have plenty of room to both ask and tell to figure out what the issue is. More than anything, this sounds like they knowingly did something shady and are now trying to shift the blame.
The OP does not give their company nor domain name. I wonder if this is related to recent efforts by the Dutch to collaborate with Cloudflare to prevent online gambling companies operating in the Netherlands.
I have been hearing stories from developers/entrepeneurs about Cloudflare being very weird to deal with:
"We'd like to talk to you about an enterprise plan."
"No thanks, I'm fine with the free plan."
"Based on your traffic, we'd like to talk to you about an enteprrise plan."
"Is there a traffic limit on the free plan?"
"No, there is no limit. But based on your traffic, we require you to get an enteprirse plan."
[Gives up and gets an enterprise plan]
[6 months later]
"Based on your traffic, we'd like to talk to you about up'ing your enteprise plan to a new monthly pay."
"Is there a cap to traffic in our current plan? I don't see that in our terms."
"No, there is no cap to traffic in cloudflare plans, but based on your traffic, we're going to require you to pay more per month than you are currently paying."
"OK, can you tell me the traffic limit in our current or new plan? So I know what I'm paying for and when I'm approaching it?"
"No. But you need to pay more."
[Wash, rinse, repeat, every 6-12 months]
It seems like while cloudflare technically does not charge for egress, in fact for large egress it's just a game of chicken between the customer and a salesperson every 6-12 months, with the salesperson trying to figure out the most they can manage to get without losing the customer? I mean, I guess that is standard for enteprise sales, but I think usually you at least have some terms to know what you've got for how long without a renegotiation?
You forget to mention that the DDoS traffic causing these issues are also behind cloudflare, but they don't give a damn about them, for obvious business reasons.
Cloudflare controls supply and demand, which, by definition of the law, should be classified as extortion.
>And then I realized that I had to hand them over my DNS? Uhh, no. It could have been "set nameserver to ours in your DNS console".
>And also there was the recent SSL spoofing they're doing even with DNS with no hosted websites. And they charge money to send a revocation.
What's your threat model here? That cloudflare will go rogue and... MITM your users? Can't they do that even if they're not in charge of your DNS? Even if you point an A record to them, that's enough to get a certificate via an ACME http-01 challenge[1].
In fairness regarding this particular post, the author admits they were probably violating Cloudflare's ToS, and they knew it.
The folks at CF could have been less obtuse in handling the matter. But at the end of the day this is an online casino breaking ToS and they got spanked.
We had a similar experience. Stuff suddenly started breaking for 10% of our traffic, support dragged their feet for weeks wrt any sort of insight as to what was going on, and then the answer was “you’re over an undocumented limit, try the enterprise plan”.
Fwiw this was some years ago and we moved most of our stuff away from them in response. I didn't get the feeling that this was malicious from their side, more like growing pains / mediocre support people / etc. But the end result was the same as you describe, except we chose not to pay up.
EDIT: more context: I shared this story on HN once before, jgrahamc responded with “please email me”, we did but it didn't move the needle. This further convinced me that CF just has a lot of stuff going on and something weird about our traffic made them error out. My suspicion is that the enterprise plan was supposed to make it internally defensible to pour more engineering resources into our case, but they were never explicit about that which made us worry enough to not do it.
I think that a large reputable business like CF should be clearer about stuff like this. That said, as someone running an API business, I also hold some sympathy for “customer does something weird an unexpected, it’s hitting a limit we didn't even know we had, srsly now what?”. The answer to that should be “work together with the customer to get to the bottom of things, customer might need to make changes too”. They didn't do that, which disappointed us, but I can relate to the situation nonetheless.
We’re still a CF customer, just not for this part of our offering.
this makes it sound like the limit is automatic or applies non-discriminately to customers, but my first instinct is that this was manually set by someone, maybe the sales reps again?
Our experience has been quite the opposite once we were forced to migrate from a free plan (a long time ago after what felt like abusing the free plan due to the amount of bandwidth we were using).
The bandwidth caps and all included features were clearly spelled out in the entetprise contract and when we went over, they didn't push for a contract renegotiation unless the overage lasted like 3+ months. And we frequently got new features included in for free.
In fact, recently they asked to renegotiate the contract due to some obsolescence and we ended up significantly dropping the bill as a result. Kind of backfired on them, I wonder if the account manager is kicking herself for this.
It's good to have an alternate experience shared, thanks!
Perhaps the stories I have heard are from people with particularly bad/aggressive sales reps, or who are particularly bad negotiators on their side.
I will say, though, that the free plan is marketted as without traffic/bandwidth limits, and has no traffic limits in it's terms of service, no? If it is possible to abuse it with an amount of bandwidth, rather than this being a "feeling", wouldn't it be more clear and transparent and respectful to just make it clear in the terms?
> In fact, recently they asked to renegotiate the contract due to some obsolescence and we ended up significantly dropping the bill as a result. Kind of backfired on them, I wonder if the account manager is kicking herself for this.
Only if the cost of supporting the depreciated feature was less than the delta.
I think it being difficult to grok is the point, if they laid down exactly how much they want you to pay for bandwidth then it would be easy to go price shopping between them and the competition. But when it's "free" bandwidth, with a fuzzy line where it stops being free, and ambiguous pricing when it does, they can hook people in with a great deal and try to shake them down later.
I still encounter people who refuse to believe that CF bandwidth isn't really free, when you can easily demonstrate that it's not by just observing who uses them. If their bandwidth truly was free and unlimited with no catch whatsoever then every bandwidth giant like Imgur would use CF, but they don't. Imgur uses Fastly, probably because it's cheaper than CFs "free".
Presumably it gives them a lot more flexibility in deciding who has to pay more.
With published thresholds they’re less able to upsell someone just shy of the limit without publicly changing the tiers. Doing that has the potential to upset existing customers who are over the new limit all at once, while also providing intel to competitors looking to undercut them.
That's not how the "free until it's not" pricing model works :P
IMHO it's just the price finding model that CF has adopted, I expect in the future they'll release limit numbers... unless they decide not releasing numbers is more profitable (i.e. the used car sales pricing model)
it's not me -- which is why i'm not nervous about talking about it on HN. I work in the non-profit sector and don't currently use Cloudflare. Just stories I've heard from others.
I'd guess that the cost of switching/cost compared to other alternatives/cost compared to business value/revenue, remained sustainable for the customer, who didn't want to deal with a switch.
In truth, this is kind of how "enterprise sales" has always worked? The salesperson trying to figure out the biggest price that won't lose the customer? But additionally having unclear terms and unclear length of contract (or really no contract locking in your terms/payment) is definitely in the vendor's favor...
Do they offer tangible benefits to justify the higher fees?
That's the thing that gets me about all types of subscriptions / pay walls. You have my attention momentarially, make your best pitch as to why paying you is in my interest.
Speaking of racketeering, it's an enlightening experience to search for "stresser" or "booter" providers (euphemisms for DDoS-as-a-Service) and look up their NS records to see who helping them ward off competitors DDoS attacks and keep their origin servers hidden. 9 times out of 10 it's Cloudflare, with the few exceptions being DDoS-Guard, who more or less specialize in facilitating crime.
I wouldn't be the least bit surprised if DDOS protection providers were using DDOS on their prospective customers via proxy. Problem, reaction, solution. Did you pay your "protection" money this month, Luigi?
This is a really important lesson here. Don't put your eggs in one basket, and if network delivery/etc. is core to your revenues and livelihood, don't trust a random third party host to look out for you.
10TB/80TB at 120k/yr, either way, Cloudflare is taking you for a ride.
If you aren't self hosting, you're really doing it wrong.
What is "don't put your eggs in one basket" here? Your DNS has to point to something... and changing it will go through propagation delays, during which it will be down if you are banned suddenly.
It's not like you can have your domain/DNS somewhere else and point to Cloudflare IPs (to not put DNS and CDN in same basket). Cloudflare does not allow that setup.
You can't protect your website from your DNS provider or hosting provider suddenly kicking you off. You are going to be offline for a couple of days.
> You can't protect your website from your DNS provider or hosting provider suddenly kicking you off. You are going to be offline for a couple of days.
Sure you can. Colocate in two or three places. You're your own DNS provider and your own hosting provider. If one of your colocation companies doesn't like what you're doing for whatever reason, you use the other two until you replace that provider.
After reading this article carefully I have a few thoughts. Firstly you were knowingly in violation of TOS after they pointed it out to you. Violating their TOS is a fast way to have your account suspended.
Secondly I'm a little confused why they would require you to pay a year upfront? I would like to hear from cloudflare as to why they required this? It's pretty fair for them to ask you to pay a year in advance because of the risk that you carry as a gambling company.
Cloudflare needed you to have to enterprise plan to remove liability from them. It's not even a big request, they have specific pricing plans for a reason.
There is a single mention of html in their terms, and it just restricts wrapping cloudflares software. Could you link me the section where they restrict anything to html-only?
It is actually more likely it is the opposite, that it got temporarily boosted to get exposure, and then fell back as interest vaned. If mods want it gone, then it will be gone.
This is the nail on the coffin, but make no mistake, Cloudflare has been a liability. It's a massive Man In the Middle decrypting all traffic, including OkCupid and 4chan for example. Imagine all those 4channers learning they aren't actually anon.
The recommendations to, basically, not keep all your eggs in one basket and have backups of config are surely good ideas. But if you have to plan on dropping cloudflare in some arbitrary 24 hour window, perhaps it’s CF that’s the problem. This sort of stuff and other recent articles about CF are so worrying that it’s now being run by the finance team (hence why every email they got in this article was from sales teams rather than any technical folks).
Also; if not registering domains on CF does anyone else do at-cost or otherwise super cheap pricing?
Wisdom of the ancients: Always make sure your domain services are completely independent from your other service provider(s), regardless of whether Cloudflare is involved. (Sorry, no recommendations at this time.)
Do we know of any alternative services providing at least the basics of what Cloudflare does?
Such as:
- Unmetered DDoS protection (i.e. no absurd base fee for it existing)
- Unmetered rate limiting (protection against cost attacks on the next)
- Reasonably priced object storage (i.e. not more expensive than numbers listed here https://blog.cloudflare.com/aws-egregious-egress)
CloudFlare does not provide unmetered anything, at best they provide services on a discretionary basis while trying as hard as possible to make it appear this is not the case. It's better to think of their product line as a CRM system with some CDN features on the side
This reads more like a shakedown than anything. Even if the casino was being dodgy, CF went in asking for more money, not demanding that they stop doing whatever it is they were doing.
This sort of situation is not time for an angry blog post. It's time for lawyers. OP needs to speak with counsel and find out whether they have a claim against Cloudflare for interfering with their business in this way. (If OP's business is so illegal they can't get a lawyer to help with that, that's another story, but it sure doesn't look that way.)
Fundamentally, the OP might be involved in something scummy or at least against Cloudflare's TOS. But if that's the case, if you have a customer who is violating your TOS, you don't hit them up and say "pay me an extra $119k a year and I'll look the other way". You say "here are your violations, fix them and prove to me that you fixed them, or pay for plan X which has terms under which they are not violations."
The way Cloudflare handled this is completely inappropriate and even if it wasn't their intention, makes it seem very strongly like extortion. Two wrongs don't make a right, and OP's business being possibly shady does not give Cloudflare license to extort them.
Way to bury the lede of OP running a gambling site and doing a ton of shenanigans to make it legal in different jurisdictions. I understand why you might think you shouldn't lead with that but it puts the entire response from Cloudflare into perspective as dealing with gambling sites sounds like a headache and it's reasonable they might not want to run that kind of venture on a regular plan.
Rule #1 in any work, business, or even personal and social life: never put all your eggs in one basket.
For CF first example, only use the cdn part, don’t use the registrar one, there are more registrars out there, don’t use their worker, use something else, and do on. It’s for obvious reasons, when shit happens, you can mitigate the impact quickly and easily compared to using all at once.
They figured out that they are losing money with you and offered to negotiate a new plan which was declined so they made and offer which was ignored and you let them know that you maybe want to change the service so they magically found an violation and after still not accepting their offer they took you down.
Pretty standard behavoir from both sides with room for improvement. They should have been more clear about what's going on and you should have been more insightful what they wanted.
In my opinion they acted to fast and not really cooperative, but if you wouldn't have declined the initial offer and started to figure out why they offer it and what options there are, you would have came out with a better deal than 10k/month and likly without 1 year upfront payment which would have given you the time needed to transition to another service.
I'm not sure. It was on spot one on the first page, then something happened and it got downranked: https://hnrankings.info/40481808/
maybe due to being flagged by people (according to another comment).
I guess it's due to general negative sentiment towards casinos, which may be understandable but doesn't (in my biased opinion) change anything about CF's behaviour in this post. I would have left it out but it's necessary in order to provide the full context.
* Cloudflare operates at a scale where its caching saves a lot of bandwidth, which saves ISPs money, which makes Cloudflare an attractive partner for peering and co-location.
* CDN is a platform on top of which Cloudflare can offer a lot of additional services that used to be expensive dedicated middleboxes.
It's unclear as it shouldn't be possible to be cheap. That said, in a world of data, Cloudflare being a massive man in the middle (MITM) probably means something [1].
[1] Cloudflare decrypts your traffic, reads it, and then forwards it. They see all encrypted data going to and from your website, in plaintext.
Im reading between the lines here but it seems like the traffic amount, the saas subscription tier, and the actions required to remidiate some issue were all unaligned.
1. Its quite possible thar CF having this site on some multi-tenant infrastructure could be threatening. Not unreasonable at least to ask them to have their own IP block.
2. If thats the issue then a clear explanation should have been provided. Routing to sales is inexcusable. Someone isnt being transparent.
3. If it’s a pure cost / revenue issue then say that, set a deadline and negotiate. This is bad karma and even though CF is clearly the market leader, what they do isnt rocket surgery. Not worth it.
My thoughts here are also all speculation, but when you mentioned multi-tenant issues my mind immediately went to a situation I've seen all too many times before:
- a companies ops team identifies a tenant that is too heavy/burdensome for multi-tenant infra and is causing issues. These issues can cost a serious amount of money if you factor in dev/ops times to resolve, other customers impacted, etc. Certainly more than what a hypothetical single multi-tenant customer could be paying
- they escalated internally and need the tenant moved to enterprise asap to resolve
- the only reason the tenant was on multi was because sales sold them the wrong thing, so now it's on sales to explain how to fix this
- improper handling internally results in this landing only on sales, with no backup, and with their task being to get them to take enterprise
- when the customer refuses enterprise they go "we've tried nothing and we're all out of ideas"
Again, this is totally speculation and I'd hope CF has more mature practices than this but this is a scenario I've seen before in much smaller orgs.
What stands out as odd to me is that CF seems to be pushing away a $10k/month customer. No business can reasonably be expected to accept sudden price changes like that, even if they'd paid, they would've moved away within a year.
Given that the article is an online casino that seems to be using potentially ToS violating domain rotation, and that they pay so little per month for apparently millions of users, I for one will not form an opinion on CF based on this article before CF has a chance to defend itself.
If you depend on one vendor, as CTO always have a plan-b prepared that you can pull out and execute. Stall, stall, stall while you're executing your plan.
$120k will never be enough, price hike is incoming for renewal.
One Question: For the Web site for my startup, I have the ASP.NET code running so ASAP will be getting into to a business account with my ISP, IPs, domain names, DNS, etc., at least for the Alpha Test.
So far, my intention is to host my own Web server. I've heard of CloudFlare, how they can help stop DDOS attacks, etc. but so far have hope not to use them.
Question: How realistic is it for me just to host my own Web server and, e.g., avoid any chances of problems with CloudFlare, the Cloud, VPNs, etc.?
In both examples you provided it was less "banned" and more "switch to a higher priced plan or we'll kick you off". In both cases they seem to be bandwidth related, and in the second case specifically they mentioned having hundreds of terabytes bandwidth but were upset for being told to upgrade to the $200 plan.
Billion dollar unregulated casino that is burning through cloudflare ips.. i don't see where cloudflare can be blamed here, they stated you should BYOIP, you didn't approve and were banned, their is no blame here.
Just FYI some countries ban casino domains/ips that are not licensed to operate even when its "just a landing page that says sorry not available"
I moved away from Cloudflare—to self hosting our network infrastructure—because, while this didn’t happen to us, I was very aware that it could. We had a great deal on Enterprise for a couple of years, but zero guarantees that it would last (and some indications that it wouldn’t). I wanted to stop praying that they wouldn’t alter the deal.
What’s especially shocking is how closely coupled sales and engineering are. Like I get they talk, but for a sales call to end in engineering pulling the plug…
To me it's pretty clear: Trust & Safety (NOT engineering; they don't appear to be involved) likely raised an alarm saying "Customer X is breaking TOS - no immediate resolution available, but something might be possible given extensive legal & engineering review. Recommend switching to enterprise so we can study how to make it work."
In that light you can see why Sales would be sent as the messenger. But I agree they shouldn't have been involved. Sending a T&S representative would have been better. But then again it looks like from screenshot #2 that they kind of did that? They just didn't have a direct call with T&S.
Another issue with Cloudflare is that it seems to be blocking VPN access to sites that have any regional restrictions.
Of course, it is easy to identify the IP addresses of the well-known VPNs, so it's not rocket science, but it does mean that popular VPNs will no-longer give you out-of-region access.
Notable, their Enterprise plan quote included BYOIP. I think that's the kicker. Cloudflare likely got a few of their anycast load balancing IPs blocked in one country, causing a huge disruption, because this customer that makes them no money wasn't in full compliance with local laws.
If that is the “key” they should be transparent and explicit about it. Now it seems CF is a Mafia that realizes one of their extorted business should be squeezed to death for more cash.
$250/month sounds like nothing at all for a site with a claimed 4M MAU. The enterprise rate of $10k/month sounds a lot more reasonable. If everything presented here is accurate, I'm not understanding the sharp discrepancy in pricing tiers. If anything they should've already been paying more than $10k/month for massive traffic on the basic plan and then be able to save money by paying for massive scale when negotiating rates for the enterprise plan.
Also this sounds like an online gambling site of questionable legality, knowingly serving customers in jurisdictions where it's illegal, so I can't say I have too much sympathy, and I feel like Cloudflare effectively fired them as a customer when they realized what they were up to.
The amount shouldn't matter, it's the unprofessional response from CF that's of concern. Also, the author doesn't necessarily say that they would be unwilling to pay more or even negotiate a higher price. The author has made it explicit that CF were more or less unwilling for any practical conversation. That, to me, is the problem. A lack of professional courtesy, communication, and transparency. Obviously there may be details from this exchange which have been omitted but if I were in this position, I would be equally upset.
I'm surprised by how many comments seem to assume Cloudflare is at fault. Shouldn't the default assumption be that no one did anything wrong?
In defense of Cloudflare, the sys ops engineer should have understood the situation and knew they were misusing Cloudflares services. They decided to play hard ball by bringing up the fact they were thinking of leaving. And we have no history of the multiple phone calls they had with Cloudflare.
While the author could improve the narrative in his article, the historical issues with Cloudflare combined with, yet another one, paint a stark picture.
Combine it with the stories I hear about Sales, the numerous other PR fumbles already mentioned in this thread, and the months I’ve personally waited (while on a paid plan!) for ticket responses only to get cookie cutter responses is, quite frankly, embarrassing.
CloudFlare puts in a good front, and their products seem decent, but they really have questionable business practices that should make anyone think twice before using them.
Naively, I imagine that Cloudflare’s math looks something like this:
(Amount owed by customer at end of month times the probability of on time full payment) minus Cost of providing service to customer for one month = profit
Since this is an online casino, could the risk of late/under/no payment be quite high?
Huh. Now that was some high-stakes poker right there. It seems the casino knew they were breaking the TOS and paying too little and Cloudflare caught up with that. Then knowing their situation they decided to ask for payment for all the expenses of the previous years (and some extra). In quite passive-aggressive manner.
But the casino still decided to stretch the penny and alas, whoever at Cloudflare was in charge got quite upset their extortion-tactic failed. So they decided to resolve it the American way and kick them out with zero warning - ouch! How fascinating.
I myself like using Cloudflare as it's quite affordable to setup and use. Makes me sad to know they have to resolve to tactics like this to finance their service. Well, at least I don't work in dubious businesses that violate TOS so perhaps I can at least wish for a graceful termination when my Enterprise bill is due.
I guess it's the natural cycle of money always spreading its tentacles to everywhere, and specifically applying pressure after sufficient metastasis and entrenching.
> Make backups of your configuration on Cloudflare. It's an unexpectedly large pain to recreate all those configurations
Better yet, configure CloudFlare through terraform, so all your config exists in your own repo all the time. It also helps day to day since it's not that hard to accidentally flip some switch in the dashboard.
But yeah, do research alternatives. CF has too much power already and will either ignore issues, destroy you, or pay lawyers to protect people trying to get you murdered, depending on their mood. There are better options.
This post was definitely demoted by HN. It stayed in the first position for less than 5 minutes and, as it quickly gathered upvotes, it jumped straight into 24th and quickly fell off the first page as it got 200 or so more points in less than an hour.
I'm 80% confident HN tried to hide this link. It's the fastest downhill I've noticed on here, and I've been lurking and commenting for longer than 10 years.
Ranking is strongly impacted by the flamewar-detector. Affected threads are automatically downranked to cool things down until a moderator steps in and manually reviews it.
> I'm a SysOps engineer at a fairly large online casino. We have around 4 million monthly active users. We had been happy Cloudflare customers since 2018 on the "Business" plan which has some neat features and costs $250/month for "unlimited" traffic.
Sorry to be “that guy,” but, you’re serving 4 million people at a casino and paying $250 a month for shared multi tenant infra, and you’re SURPRISED you have problems? Really?
To be honest, I’m glad these sorts of businesses get kicked off Cloudflare because it causes problems for others sharing the same IP space and infra. I’ll let someone else with experience discuss how many times a day the network would see a hacking or DDoS attempt against the online casino, which is by far the favorite target of hackers. But in general, I just don’t want any of my infra touching the same stuff as these guys.
Like another person here, I am assuming that Cloudflare ops told someone “tell these guys to get their own IPs and upgrade,” and then the message went to Cloudflare’s (utterly lousy!!!) sales people to try to fix before shutdown, and then it all turned into the mess we see here.
The true moral of the story, I think, is, if you’re running an online casino on a shoestring budget, expect bad things to happen to you. Of all kinds.
Indeed. Based on what has been shared by OP, they could have a case.
If OP’s business was in fact illegal, CF should have stated it. Now it seems CF is an evil sales driving monster. A monster that grew so big it thinks it can do whatever it likes.
The sad part is that, assuming OP is not leaving out critical parts, multiple people play parts of this evil machine. I’ve seen how sales people think. But this is next level toxic culture. The second customer threats of leaving for the competition, they freak out and pull a bigger lever to destroy them. And the fact that a company allows this to happen…
I would never do business with CF. Good thing i don’t right now. Cause i will definitely take it elsewhere.
> This could arguably be seen as a violation of the Cloudflare TOS, as they wrote above.
And the very next paragraph begins with:
> In any case, we receive >95% of our traffic through the main domain that’s been unchanged since our founding, and were happy to resolve this issue in whatever way...
And then they complain about paying up?
The only issue I see here is around the aggressiveness on the CF side. But, I was not in those meetings and the way above reads tells me that I might have been slightly mad so perhaps the CF was just taking it out on them?
Sidestepping the whole ethical conversation and just taking it as a good action for the hosting provider to do it still fails the sniff test as Cloudflare (according to this story at least) didn't seek to take it down rather sought to make more off of it.
After they deplatformed KiwiFarms, I thought that's an isolated case but turns out they are just unprofessional. I can't have pity for a casino service anyways.
> a web forum that facilitates the discussion and harassment of online figures and communities. Their targets are often subject to organized group trolling and stalking, as well as doxxing and real-life harassment.These actions have tied Kiwi Farms to the suicides of three people targeted by members of the forum.
CloudFlare only dropped Kiwi Farms because Keffals made it inconvenient for them to do business with enterprise customers. Said customers were looking at Twitter and saying, "Wait, you host WHAT?!"
Before that CF was high and mighty on the "free speech" horse.
There's an old spat between CloudFlare and Malwarebytes where MB was threatening to block all of CloudFlare because they wouldn't remove literal malware. The argument being that running a reverse proxy "isn't hosting", and should be treated differently, even though to literally anyone else there's no difference between an origin server and a proxy.
CloudFlare is just sketchy as all get out, IMO.
Putting my biases on the table: I think CloudFlare shouldn't have hosted Kiwi Farms in it's current state, because I don't think hosting dox should be legal. A website that hosts dox is not engaging in speech, it is engaging in censorship. Hell, in the EU, it's already illegal to host dox, the US just needs to pass a privacy law comparable to that of the GDPR. Kiwi Farms is legal in the US purely for the same reason why the CIA/FBI/NSA can legally buy advertising data from Google and Facebook.
Yes but then people argued it's about these websites being bad. Turns out it's just about money, considering Cloudflare has no problem with CP or other website calling for violence and celebrating it.
Cloudflare is evil, even though they do a decent job of pretending they care. They are smart to offer gateway services (gateway as in gateway drugs - stuff to get you hooked) for free to end users, since it grows their fanboi base, gets more people familiar with their services, and gets people hooked in ways that make it very difficult to use something else when their projects grow.
However, any reasonably competent person can see that recentralization of the Internet is a Bad Thing™, and that this is precisely what Cloudflare wants.
Likewise, we know that aggregating our data through a for-profit company that's based in the United States means that collected data is reasonably in the hands of the NSA, which makes their DNS-over-HTTPS scheming suspect.
Just like what happened with the company in this post, we have plenty examples of them abusing their position to extract money from both legitimate companies, like this one which is aware of their legal obligations in various countries, and scammers and spammers alike, who Cloudflare are more than happy to host indefinitely in the name of "free speech".
Their lack of clear communication, their broken abuse reporting, their continued claims that they don't "host" all show them to be antagonistic towards anyone negatively impacted by their facilitation of illegal activity.
Cloudflare is an evil company that just happens to be better (but not great) at hiding it than other evil companies.
The consistent reaction when I brought up Cloudflare with other CDN technical and sales teams, as of about four years ago, was a laugh and something along the lines of “yeah we’ve got some customers with some stories about them”.
Bait-and-switch seemed to be the most common pattern, plus crazy-high prices once you’re on the “switch” side of things.
But their sales team was so uniquely uninterested in our business that I never had to find out first hand.
> However, any reasonably competent person can see that recentralization of the Internet is a Bad Thing™, and that this is precisely what Cloudflare wants.
It's an inevitable outcome, as long as there is nothing done against the big threat actors: government-run APTs from China, Russia, North Korea and Iran, government-tolerated scammers (India, Turkey), rogue actors in our governments' security services (e.g. Pegasus), ordinary criminals mass-hacking vulnerable devices and selling access to them to be abused for DDoS'ing for less than the cost of a coffee at Starbucks... it's a wild west, and people are hiding themselves behind the largest giants they can find: Cloudflare, Akamai, AWS, Azure and GCP.
You're throwing unrelated facts at the statement. Recentralization has absolutely nothing to do with being protected from DDoS or from other threats. Now, DDoS (and other kinds of) protection can be done by recentralization, but that's just one possible way.
Saying it's inevitable makes you seem like a Cloudflare apologist, which unfortunately we see way too often here on ycombinator. Has anyone refuted my suggestion that Cloudflare is knowingly evil? No. Have they downvoted because my information is incorrect? Also, no, or if they think I'm incorrect, they haven't bothered pointing out how.
People want to like the things they choose, and this, unfortunately, is where Cloudflare is cleverer than other large, evil companies.
As far as I can tell, the issue with this is:
OP runs a casino/gambling site. Gambling is a regulatory mess (I have spent far too long dealing with this as an RNG supplier), and so it's very hard to comply with every jurisdiction, and each one needs you to prove compliance to operate in that jurisdiction.* Gaming companies spend a lot on compliance and tracking, but since the internet is the internet, it's pretty hard to enforce perfectly, so some countries and ISPs take this into their own hands.
Due to that, IPs hosting gambling and gaming sites often get regionally blocked by internet providers or otherwise flagged as hosting illegal content. Those regional blocks consequently affect the reputation score of the IP, and if you are a traffic aggregator like Cloudflare, can cause other customers to have issues. One of the most aggressive and annoying regulatory environments for gambling companies is the US, so it's very possible Cloudflare has had some trouble due to gambling use of their IPs in states in the USA.
Cloudflare wanted them to use the BYOIP features of the enterprise plan, and did not want them on Cloudflare's IPs. The solution was to aggressively sell the Enterprise plan, and in a stunning failure of corporate communication, not tell the customer what the problem was at all. The message from Cloudflare should have been "Enterprise plan + BYOIP or ban, and maybe we'll work with you on price" but it was instead "you would really like the Enterprise plan."
*As an aside - we're lucky in that respect being a tech supplier with relatively uniform rules, but our customers (the gaming companies) get the short end of the stick here.
BYOIP is reasonable, though I doubt anyone actually does legislation blocks by IP. Since like half of companies on the internet use Cloudflare or other multi-tenant infrastructure everyone is aware that you can't block an IP address and hit one target. The only thing I've seen is DNS blocks (both DNS protocol directly and based on TLS SNI).
FYI, we also fully block users from the US (due to regulations).
My problem here is mainly the unprofessional communication and huge mess of mixing "compliance" with sales, without giving any clear information or options. And then the removal of our account without warning while we were still talking to them.
You would be surprised how big of a hammer ISPs will use when they are told to hit something. They live in a very different world than many modern web software companies - they are the plumbers for lots of things you take for granted, and look at the world the way a plumber does. Thanks to TLS, the plumbers can't see the HTTP headers to figure out what's actually flowing, so they sort of end up whacking all of it.
Generally, low-reputation IP addresses are associated with scams, spammers, and other similar things. Gaming somehow gets lumped into this bucket in some jurisdictions, but that hurts you worldwide (similar with other "sin businesses" like porn). These blacklists get published (I think there's some parts of BGP that make this happen, but I'm not quite sure what the mechanism is), and being on any one of them hurts your traffic everywhere because it becomes suspect.
I agree with you that this mix of compliance, engineering, and sales is gross. If this was the issue, they should have just told the OP.
2 replies →
While they absolutely shouldn't ban IP for reasons you said, some do that anyway.
The most (in)famous case is China's GFW which banes IPs all the time. Yes, other websites often get accidentally blocked, but they don't care. Moreover, you can't even communicate with them because there are no official legal regulations. This is something what any CDN or cloud providers have to deal all the time.
2 replies →
>The solution was to aggressively sell the Enterprise plan
A demand for 120k upfront or else bad stuff happens is by no reasonable definition "selling", aggressive or otherwise
Sounds like something Sammy The Bull would be sent to do, not the sales team.
I also wonder if the company in the article didn’t know that (either by reading between the lines as you did or via other correspondence they didn’t mention) and weighed that in their decision to go with Fastly.
BYOIP isn’t just expensive—if your content is bad for IP reputation the time-to-flagging of your IPs is going to be way shorter on BYOIP than on shared IPs due to there being less dilution. And that’s without getting into the challenges around rotating/renting IPs on a continual basis.
I do agree that CF did not communicate that well or professionally—if the sales emails are the only comms that happened here.
I think it is possible that the company posting this didn't realize that this might be the issue, but you are right that they may know. It may have been a small company, even doing that much bandwidth. Online gambling sites tend to push an entire video game when you are playing on their site.
Many gambling companies are fine just doing BYOIP or running dedicated hosting infrastructure that is on providers who are explicitly running hosting for that industry (although they are moving to cloud). There is a good reason this separate infrastructure exists. In general, I would not assume they are rotating IPs: this is not a scam, it's a business, and they are largely fine with being blocked in places where they can't legally operate.
Maybe the first couple of weeks there was a misunderstanding, but by May 7 it was clear what the situation was. They was told they need BYOIP through and enterprise plan; they just didn't want to pay for it.
>I have spent far too long dealing with this as an RNG supplier
Is there much of a market for that? I thought random.org had it all sewn up.
No, random.org isn't in that market at all, aside from doing some drawings local to them and unofficial games.
I think we are only one of ~3 TRNG suppliers who have been audited. Many games don't use a TRNG, though.
Since it uses atmospheric noise, you can also influence the numbers from random.org by transmitting radio waves in the area nearby - the operator of random.org has mentioned that there's so much RF activity that he is concerned about whether the bits are still random. A final issue is that they are also so low-volume that they probably can't get enough test data for the required audits (which can be a lot of data).
To underscore the volume question: Random.org used to have a running count of bits generated. The counter wasn't monotonic (before it broke in ~2015-2019), but the peak value I saw when I checked archive.org was about 250 GiB total since 1998 (that was in 2015). That is one quarter of the size of our "light" qualification test ("heavy" is 16 TiB). The RNG auditors also take O(100) megabytes for each audit, which would be a significant fraction of random.org's output.
1 reply →
> Is there much of a market for that?
I don't think it's a huge market, but state-run lotteries around the world need good random number generators for games without physical balls (like Keno for example).
I've talked with people that have created RNGs (rather than buying off-the-shelf solutions) and it sounds like soul-crushing work - mostly due to dealing with the government regulators that need to give final approval before the RNG can actually be put into production.
Gaming machines are highly regulated, almost like medical devices.
There are seals on the hardware, any modification must be approved, you must certify that the payout is the expected one, ...
6 replies →
Ok, I have to ask: are you a Random Number Generator (RNG) supplier, or a Renewable Natural Gas (RNG) supplier, or some other kind of supplier??
Considering they mentioned working with gambling/casinos, I would assume random number generator. Which may seem somewhat trivial to build a business around, except if you’re in a highly regulated industry like gambling that regulates the implementation of randomness (and probably requires auditing and other complicated things like that). I would love to read a blog post on all the complexities at here.
It almost sounds like you are excusing them. Asking people to switch to the enterprise plan and bring your own IP is reasonable, but not on a timeline of 24h, and trashing their account when they tell you they are talking to a competitor makes me feel like I should flee Cloudflare services with all haste.
> and trashing their account when they tell you they are talking to a competitor
That is just a story they have made up. They don't know why Cloudflare shut their account down. I reckon the Fastly "reason" is likey a red herring.
1 reply →
This is just how people communicate now in the business world. It's up to you to read between the lines.
One thing I've learned to be wary of on the job is "do you need help?" That phrase is often code for "You are not performing up to our expectations. This is your first and only warning. Get in shape or get out."
Poor communication is worse than no communication.
[dead]
The way Cloudflare approaches situations like this is not ideal for anyone.
You start using the service and don't pay a lot, so you make plans around a certain level of expenses. Then out of the blue you receive an "urgent" email from a sales representative and suddenly you have to go from $20 or $250 to $thousands right away.
Obviously it's not in CF's interest to keep a customer that doesn't pay enough, but dropping a "bomb" on the customer and make them feel like they're about to be kicked out from the service makes the customer lose trust on CF.
CF can probably match Fastly's price. If they had acted differently in this and other similar situations, they could keep the customer, be paid more, trust wouldn't be affected, and there would be no bad PR here.
Since the CF management that posts on HN usually say this is not supposed to happen, perhaps someone needs to sit down and look at the incentives sales reps have? Even if you don't care about the customers, this is affecting the CF brand a lot.
> but dropping a "bomb" on the customer
Why on earth any company would jump from $250 to $10k per month unless they had a gun to their heads? Even if their revenue is to the millions/billions (which most likely is considering the nature of their business). They work for their own profit, not Cloudflare's.
No one wants to pay more, but if they do it in a way that makes their customer run to a direct competitor and not want to come back, then maybe there's something wrong with their approach.
We only have one side of the story here, but it's not the first time I've seen posts/comments about these emails from Cloudflare and the messy communication that follows. As a business customer, I really hope I don't have to deal with any of this.
The point is it doesn't have to be a "gun to their head". It just needs to be a serious of emails, calls and negotiation.
I'll be they are now paying Fastly a lot closer to $10k/month than $250/month
Seems pretty reasonable if the $250/mo plan is costing Cloudflare close to or more than that amount of money due to any loophole or other unforseen expense in the plan.
What seems interesting to me is just what the loophole is and how many other business are also on the radar for this drastic pricing change. Are there other goodwill discounts Cloudflare is ready to start collecting on, or does the gambling site represent a unique situation?
have you seen the twitter post from Gamdom about mentioning the fastly quote in negotiations? they were taken down instantly. It doesn't sound safe to mention competitors with them or even exercise the prospect of leaving
I will remind HNers: is Cloudflare not the company that leaked sensitive data through cache files that were indexed by at least Google, and when the tech community were up in arms about the massive leakage of sensitive data, the CEO’s strategy was to turn up here and criticise Google for not deindexing quickly enough?
You get what you pay for.
That's one of the main reasons I'm leary about them. Such a big f-up is difficult to forget. It shows that they have a move fast and break things culture which for a company that is responsible for critical infrastructure feels wrong.
In response to this incident Cloudflare has made big engineering changes, including huge work to move away from C as much as possible.
The offending parsers were rewritten in Rust (https://github.com/cloudflare/lol-html), as well as WAF, image optimization, and a few others. Nginx is being replaced with a custom cache server.
New implementations are using either the Workers platform, or are written in Rust or Golang.
2 replies →
I interviewed there once and they asked me what I would do if a service broke after a deployment. I said the first step was to revert to the last known good version and then investigate. Color me surprised when that was not the answer they expected.
6 replies →
I remember them criticising Google for not being faster at removing cached files. I don't remember them blaming Google for their screw up.
And let's be honest, if a big provider wants to offer cached versions of pages, they probably should have a way to purge those files in case there's a problem (eg: malware).
> I don't remember them blaming Google for their screw up.
You're putting words into my mouth.
13 replies →
This was a much needed reminder. Although, it's quite difficult to find a better DDoS mitigator which is better than CF, I still wouldn't trust them for everything. Especially, since they are most likely snooping on the decrypted HTTPS connections
> Although, it's quite difficult to find a better DDoS mitigator which is better than CF, I still wouldn't trust them for everything.
Adding Challenges, TLS fingerprinting and Rate Limiting is possible on just about every major CDN platform to be honest. I guess with CF it's more "ootb" though, where you don't really have to think too much about policies - but at the same time, you can't go as granular in those policies (e.g layered) as some others.
At the moment the account got banned, I would guess that the CloudFlare sales team had this down as a "60% likely to close, estimated close in 6 weeks".
There is just no reason they would suspect that they were going to lose the deal to Fastly at this moment. They were very much the default winner.
Extortion or not, I just can't fathom that they ragequit the deal at this moment, because they were about to win it.
It therefore seems likely that after looking into it they disqualified it as a business category which is against their TOS or whatever.
Or that the enforcement and sales teams have very similar, overlapping triggers for engagement, etc.
Cloudflare's behaviour here was shitty and this is not the only report. By all means their reputation is very generous free tier and a horrible experience in paying.
BUT seriously who ragequits a winning deal? Another comment summed this up - the attention caused them to take a look and realize they don't support shady-ish casinos, possibly (seeming to) evade US legislation, etc.
> It therefore seems likely that after looking into it they disqualified it as a business category which is against their TOS or whatever.
The first sales email is from a Cloudflare with “Gaming Division” in their email signature, so they were clearly aware of the nature of the customer’s business. Moreover, it seems they have an entire department dedicated to serving the gaming market.
It's quite a common pattern in saas. Someone gets through automated ToS checks with some niche use case and escalates some unrelated issue to support which triggers manual ToS review. That being said, a corporation as big as CF with 120k usd bills should do better and never let this happen. Very amateur.
Sounds like OP is a casino and plays domain games to avoid regulatory interest. Recommend reading article carefully before reacting to the headline. Hopefully Cloudflare provides a perspective.
Hmm. My take is the casino structured its business to comply, not to evade interest. Further, I don't see how Cloudflare benefits by taking on the risk to charge more to help a customer avoid scrutiny. More like: they know it's a humming business and want a piece.
The way I read the screenshots of the emails from the articles seemed to suggest that something the authors company was doing was causing issues with IP reputation on CloudFlares range.
Them very aggressively highlighting the BYO IP feature and then even suggesting third parties to rent IPs from strikes me as a significant detour from their normal “script” (having dealt with their AU sales team before).
11 replies →
Compliance:
> We do have multiple domains that mostly act as mirrors to our main domain. We have these for a few reasons. One is that since we are a casino, we have different regulatory requirements we need to comply with in many countries.
Evasion:
> Another is that we use them to target different global user groups and affiliates and track conversions long-term. This also means that if a country DNS-blocks our main domain, a secondary domain may still be available.
This is more like one gang hitting up another for "protection" payments. I had to laugh when they called it "Trust & Safety".
> My take is the casino structured its business to comply, not to evade interest
It's impossible to say what's going on since it's an anonymous post with no details.
Maybe it's all 100% true.
Maybe there are some key details being left out. Wouldn't be the first time I've seen one of those outrage posts that seriously misrepresented things.
Whatever the case, obviously the author is not an unbiased party. These posts do well because "zomg Cloudflare bad!", and maybe they are, but I sure as fuck don't trust some casino guy either.
1 reply →
But for $10k a month cloudflare is ok with that? Either it's acceptable or it's not, there is no way that this looks good for cloudflare either way.
A reasonable scenario to me seems to be: An automatic "upgrade to the enterprise plan" requirement was triggered, and then in the process of the sales calls to make that happen, Cloudflare got serious eyes on the customer for the first time (whereas at a paltry $250/month previously they wouldn't have), and realized exactly what line of business the customer was involved in, and decided to fire them.
6 replies →
If it's legal but burdensome (somehow) to host a particular industry, requiring more money to deal with the increased burden seems reasonable. For instance, if their legal department needs to deal with complaints from various countries, that probably costs more than $250/month.
That being said, I doubt that's the core issue in this case.
That isn't how the world deals with risk.
If you think something your client wants could explode into a liability, you can turn them away or you can just make sure their bill covers your exposure.
If it's a legally questionable service, there's likely to be plenty of abuse contact, or they're going to be a big target of crime, they're going to end up paying more. This is the same reason why some industries (eg porn sites) have always paid more for card processing.
It's not just 10k a month. it's 10k a month for the plan that allows you to BYOIP (Bring your own IP addresses). That was cloudflare's issue.
Their business was causing IP reputation damage and all plans but the enterprise BYOIP plans share the same IP pool.
Essentially it was "use your own IP pool and pay us for the cost of maintaining that pool for you or GTFO".
This wasn't just a normal sales rep hitting them up. This was trust and safety (i.e. the moderation team) coming to them with a compromise that would allow them to stay on the platform. They chose against that and were dragging their feet.
The timeline of the article also really makes this clear. This wasn't over the course of 24 hours. This started a full 4 weeks prior with sustained back and forth. They only included a few images of emails from the discussions but the article makes clear that there was more discussion happening.
And to quote the article. After receiving the ultimatum, they got an entire extra week to deliberate.
> We managed to buy a week of time by letting it escalate to our CEO and CTO and having them talk directly with Cloudflare.
Then finally when they told CF that they were just buying time while looking to move elsewhere, CF dropped their act of goodwill and the moderation team resumed the moderation action they would have taken in the first place had this been a smaller account.
----
So yeah it sounds bad from the snippets but this was basically "hey you are a big customer and you are breaking rules we would normally ban anyone else for but if you can compensate us we'll spend the labor hours and infra to let you keep operating in your own little quarantine box.". So this really should be seen as an act of goodwill rather than malice.
5 replies →
I can reason my way into it, I think objectively. To protect their IP reputation, CF required BYOIP. This costs them something, and de-jure requires an Enterprise plan. Which for the customers usage costs $X. Is it right? Ehhhhhh. Does it follow corporate logic? Yeah. (Sales logic? YES)
I'm not defending Cloudflare's exact actions in this scenario, but it seems reasonable that there are cases where yes, for $10k Cloudflare is okay.
Risk can be mitigated, especially if you take care to know what the risk is, but risk mitigation and the salaries of the risk mitigation teams are not free.
The answer of "no, we will not host you unless you pay us enough money to hire people to make sure we're not breaking laws by hosting you" makes plenty of sense, and an online casino that is likely dubiously legal in many countries is definitely a place where you might use that answer.
I'd also expect there are cases where Cloudflare enter into enterprise agreements with customers, get a good hard look at exactly what's happening, and then tear up the agreement and walk away.
2 replies →
That's not true at all. That line of argument gets close to "if this product is free for open source, why is it not free for me? either it costs something to operate or it doesn't." You don't get to price the service.
1 reply →
The point is more that the author is an unreliable narrator and you need to apply a little salt to the rest of the story. Cloudflare absolutely shouldn't be taking bribes to permit regulatory evasion. But if they are, I want more evidence than a substack post.
11 replies →
Taking a step back, why would they even care if their platform is supposedly neutral and not responsible for the content ?
If they can indeed stop providing services to a casino, why cannot they shutdown a website spreading pro-war propaganda, or a website selling illegal services ?
It means they are making editorial choices, instead of just being the technological provider and being a neutral "internet pipe".
Not sure it's really in their best interest to self-police in the end, as they could lose their DMCA safe harbor provision ?
> Taking a step back, why would they even care if their platform is supposedly neutral and not responsible for the content ?
Because their main network all uses one big IP address pool and the blocks by various regions/countries against their site were probably not just DNS blocks but also IP address blocks.
So they now have an account whose activity is getting their IPs banned in countries where they operate.
So they told the account owners they needed to pay for an enterprise account and a dedicated IP address pool maintained by cloudflare. That's why CF kept talking about BYOIP in the emails.
i.e. "Pay for us to build you a quarantine with your own IP pool or leave ASAP"
1 reply →
> Not sure it's really in their best interest to self-police in the end, as they could lose their DMCA safe harbor provision ?
This.
That said, we're seeing this across so many platforms, from datacenters to social network sites.
They blew their safe harbor provisions years ago and yet remain untouched despite this.
1 reply →
I do encourage you to read the whole article cause there is some fine details in there. The main point is that we were happy to remove any domains apart from our main domain (which gets > 95% of our traffic) but Cloudflare did not give us that option or any other detail on the supposed issue.
If 90 or 95% of their traffic comes from a single domain (and presumably has for a while), that still doesn’t make OP sound guilty. If there was a legal issue Cloudflare legal should’ve stepped in, not their sales team.
That was the part that bugged me. This workflow is very busted from a user standpoint, though I'm sure it works very nicely to Cloudflare!
It smells like the "problem" was detected by automation, but instead of being able to reach anyone technical to work through it, you can only call sales teams.
In my opinion it's one racket vs another.
This shouldn't matter, in general Cloudflare responds to complaints about allowing illegal content with "we're a neutral utility, we forwarded your complaint to the site's webhost". To me, the article showed that Cloudflare was being extremely aggressive with selling the customer on an enterprise plan and repeatedly invented excuses to get them on the phone with their sales team. They then took the site offline and locked them out of their account when the customer started talking with other CDNs.
So the thing that stands out in the article, is that cloudflare's initial communications (and the final communications, when they moved to ban) implied issues with their behavior (trust and safety team, terms of service violations), but in between it sounds like the didn't talk about ToS at all, just sales team asking them to buy enterprise. Though it's possible OP is omitting some explanation given by as why enterprise plan would alleviate ToS issues.
>Hopefully Cloudflare provides a perspective.
Well HN is the unofficially official Cloudflare Support forum. I think we will hear from them soon. From past experience normally their response time for anything Cloudflare on HN is within 2-3 hours.
Except Cloudflare position here is not to ban them but they want to get paid for it. You are shaming the OP and his business but the reality is that Cloudflare has acted in a worse manner and that should be highlighted.
How does paying $10k a month solve that?
For $10k / mo paid 1 year in advance, your cloud provider does a legal review of the situation and figures out how to make your problem work on both the technical and legal level. It's not a "special plan", it's consulting.
Edit: "How do you know?" -- I don't know it's actually what happened, but when switching to enterprise, you don't go from 10% margin to 98% margin. The added costs actually represent added budget for the provider to deal with your "special case". ALL enterprise pricing tiers are disguised consulting contracts.
2 replies →
It's 10k a month for them to set up a dedicated IP address pool so that they could BYOIP and buy their own IP addresses instead of getting the IP addresses in cloudflare's main IP address pool repeatedly banned or reputation harmed.
i.e. it's a $10k fee for maintaining the infrastructure for a quarantine around their services
5 replies →
Nice place you got here. It'd be a shame if something happened to it.
Except the "place" isn't Mom and Pop's bodega, it's a casino dodging countries blocking its main domain.
2 replies →
This is literal theory crafting lol. CloudFlare never said or implied that in their emails, yet you seem to know more than the CF reps themselves?
"Now this needs a bit of context on what they are talking about. We do have multiple domains that mostly act as mirrors to our main domain. We have these for a few reasons. One is that since we are a casino, we have different regulatory requirements we need to comply with in many countries. For example, many games are only available in some countries. Some countries we block completely. Then we have a few different domains that remove certain game groups or site features - for example our social features (chat, user tipping / interaction) or our sportsbook. Another is that we use them to target different global user groups and affiliates and track conversions long-term. This also means that if a country DNS-blocks our main domain, a secondary domain may still be available. This could arguably be seen as a violation of the Cloudflare TOS, as they wrote above."
Looks like they COMPLY with regulatory interest, to me.
When it comes to laws and taxes, "comply" and "evade" tend to be synonyms.
"In order to comply with tax regulations and donor laws, we had to structure our activities in order to make it possible for political donations to be classified as regular consulting income".
2 replies →
If that's the take, that means Cloudflare is okay with 'breaking laws' so long as they can take a heavy cut of the ill gotten gains? </sarcasm>
Let's not try to find reasons to harm the messenger and stick to the facts -- a paying customer was suddenly extorted for hundreds of thousand of dollars out of nowhere.
Using localized versions of your services to comply with regional laws and enhance user experiences (i.e. make money) is SOP for practically every international $bigco. Online gambling is regulated and legal in ~50%-70% of the world; without actual evidence to the contrary, it’s completely reasonable to assume that this is a legitimate business. I’m really struggling to agree with the “two sides to every story” replies being left here about how there’s likely shady activity going on behind the scenes, when to me the post read as candid and transparent about the nature of the nature of the business, the admitted legitimacy of CF’s TOS violation claims against them, and the content of the communications with CF.
My 2c: It’s scummy that CF did this. It looks like they were disingenuous about the severity of the violations and used it as an excuse to get more $$$ from an already paying customer to make the manufactured problem go away.
It is a good article, good to have practical details of how this goes down... but really an international casino cant afford more than $250 a month?
Getting a demand to increase the payment by 40x is shocking no matter how much you make.
1 reply →
Nah, you have different domains so you can track and maintain flows, also the regulations might even stipulate having domains in the locale, the headline is very much accurate after reading the article.
I mean, sure, they’re probably doing some sketchy regulatory dodging or whatever. Which part of this can Cloudflare solve by having them pay $120k/year to them?
The post mentions BYOIP. I assume Cloudflare wanted OP on BYOIP to mitigate risk, and Cloudflare wanted them to pay for the privilege.
The part where Cloudflare is happy to turn a blind eye to any “issues” if they get their $$$, apparently
Over-charging is a legal way to effectively deny service. When I'm offered jobs I don't want, I sometimes tell them my salary is 3x what I really need.
1 reply →
With a casino, the issue isn't just domains, it's also IPs. That's why they pushed BYOIP so heavily.
Obviously you work for CF. I did read the full post.
Yeah, there's some pretty key info being left out. I don't doubt that Cloudflare communication sucks especially when dealing with their sales team (aka bizdev which is what OP was originally contacted by), but the second screenshot is pretty damning.
My guess: Their account fell out of the non-enterprise TOS for some reason which is being obscured in the post (probably domain rotation related). Their T&S team proposed moving to enterprise for a custom resolution. OP's company refused, their account was purged because they had gotten several warnings about it.
I'm sure this sounds frustrating to the average HN dev who runs a legitimate startup with cloudflare on top and is now biting their nails worried to death about what will happen to them. But "online casino" immediately raised a million alarm bells in the post.
I did mention the multiple-domains issue in the post. It would not have been amazing for us to remove our secondary domains, but we would have been very happy to do it if it had resolved the issue. We asked them again and again but they would not give us any detail or options apart from their 120k/year package. Note that BYOIP (which I guess they could reasonably have required to isolate us even if we only use a single domain) is available for a fraction of the cost elswhere (e.g. fastly).
Since we already left Cloudflare the only reason I finished writing this article is to warn others. I think it's still relevant to many companies regardless of what you think of casinos, since very unprofessional sales tactics (unprofessional as in business threatening) seem common place with them. Do look at the linked other posts and comments here from other people affected that don't have anything to do with casinos.
I'm happy to answer questions as well.
1 reply →
And the part where they offered to remove the secondary domains and couldn't get an answer?
Is the casino illegal in the jurisdiction they're based out of?
It doesn't seem so, so there is at least a valid reason for Cloudflare to keep them as a customer as they're not violating the laws where they have their business in.
This has been my experience with 80+% of these loud complaints about services, especially regarding "losing Google traffic". Dig into it just a little and you find out the complainer was doing something extremely shady that the service is often too polite/proper to call out in a public forum.
Cloudflare was the company that went viral for the firing of an account rep not hitting her goals. I wonder if it’s overall indicative of not a great culture in terms of relationships with enterprise customers.
https://m.youtube.com/watch?v=7LuwPdp-_4c
> Cloudflare was the company that went viral for the firing of an account rep not hitting her goals.
How is that something worth going viral over? Salespeople get fired all the time for not meeting their sales goals. Engineers similarly get fired all the time for not meeting their productivity goals. If you don't do your job well, don't expect to keep it.
And if I recall correctly, in this particular case, she was a green employee who hadn't even made a single sale yet! What more obvious of a layoff target is there than that? Would you keep a green unproven salesperson over a proven veteran salesperson who's landed 9 figures in sales?
While I agree with you, I think the call is for companies to be less psychopathic and stop onboarding people within 90 days of mass layoffs.
Especially in a world where people pick up their whole lives and relocate for jobs. Recent joiners aren't getting any sustainable kind of severance either. The idea is if you're hiring them you have a minimum commitment to support their success.
Yes she was an obvious fire, but it's also the organization's fault. Enterprise deals also take way longer to close than that...
All that said, salespeople can and do move jobs a lot. I'm sure she'll be fine.
7 replies →
She did say that she didn't bring in any customers...
She did say she was around for only 6 months and enterprise sales cycles can last 12+. Though I guess if you’re engaging in scammy behavior it can be much less… maybe she wasn’t willing to do that.
6 replies →
We had a site hosted on CF business plan with fairly large bandwidth usage (completely legal, had a lot of media). They approached us with an enterprise plan but we did not have the budget for it.
Asked for a little time, they said fine and we moved much of the bandwidth usage to a couple of dedicated servers on OVH I think.
Never heard from them after that.
How do you deal with DDoS attacks against said OVH servers?
> By default, every OVHcloud product is supported by the Anti-DDoS infrastructure to defend against malicious activity. https://us.ovhcloud.com/security/anti-ddos/
3 replies →
Can I ask how much bandwidth we are talking?
We are doing about 3TB
This was a couple years ago so not sure but likely 50+ TB/month.
I have been moving 1TB/mo on a free account for personal stuff. So far so good.
Can any1 explain how HN algo works that this post, which at time of writing has 355p, 180comments while being posted 1 hour ago, isn't even on first page (ranked 31)???
It set off the flamewar detector, got flagged by users, and got downweighted by a mod.
The 'customer support of last resort' genre is common and not usually a good fit for HN [1]. If people feel this story is unusually relevant and interesting, I'm not sure I agree—long experience has taught us that one-sided articles like this nearly always leave out critical information—but I also don't mind yielding in an occasional specific case, so I've rolled back the penalties on this thread.
The issue from our point of view is not about story X or company Y—it's a systemic one: the most popular genres of submission (especially the rage-inducing ones) get massively over-represented by default, so countervailing mechanisms are needed [2] if we're to have a space for the more intellectually curious stories that the site is meant for.
[1] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
[2] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
I would be very happy to hear Cloudflare's actual side of this. (Or - it would have been great if they had given their side to us before getting into this mess). The only critical information from our side that I'm aware of is that we're a casino with multiple domains - which is why I put that right at the top. But most of the info should be relevant to any business interacting with CF.
I do admit that I originally drafted this article as a "customer support of last resort", since that seems to work well for CF specifically. But it's too late for that anyways by now - the problem is "resolved" by fire and we don't plan to move back.
I purely posted it now as a precautionary tale for other people because of all the pain it has caused us. So the audience is tech people in most companies of small size that will hit more traffic at some point in the future.
4 replies →
> The 'customer support of last resort' genre is common and not usually a good fit for HN
They're already off Cloudflare, I would see this story more as "Dealing with tech company X is a business risk" cautionary tale.
2 replies →
if you check sites that track HN's rankings, it was ranked #1 for a while, then it suddenly dropped to #27 and continued declining https://hnrankings.info/40481808/
thanks for sharing. I didn't even know this was a thing! Comparing to 4 or 5 others, this definitely looks more like a step function into obscurity unlike the other lol
yes that's pretty blatant isn't it
My guess for why this would be flagged: its a gambling website so people are unsympathetic. Definitely an abuse of the flagging system.
Maybe HN needs a flagging ring detector as well as its voting ring detector.
1 reply →
Wonder how many of the people flagging it work for Cloudflare?
I though flagged content showed a "[flagged]" in the title?
1 reply →
Flags below the [flagged] threshold are invisible but still act to downweight the post.
I've noticed that cloudflare complaint threads get flagged with surprising and unwarranted regularity.
5 replies →
Yeah I commented, refreshed and was shocked to see it disappear. It's ironic that we're (reasonably) asking for transparency about a post which is kind of about non-transparency.
It could have flags on it or maybe the flamewar detector got tripped.
Moderator action seems unlikely because it’s still on the second page.
Remember, on HN there is such a thing as commenting too fast
If a thread is too interesting, it gets penalized, can't have too many people commenting on an exciting topic
1-The gambling business is shady by design, whether you like it or not. This was probably more risk than benefit for them based on what they were getting from you.
2-Your business is probably very profitable, and $300 a month is very cheap compared to the potential hassles they could face working with such a business.
3-I find it very inappropriate to dox business representatives and show names when you have carefully hidden any information regarding yourself and haven't even disclosed your company name.
After all they can choose with whom they want to do business. They gauged what price they could ask you, factoring in how profitable your business is and how noisy and painful it might be to work with you. It sucks but this is the downside of SaaS/PaaS.
Gambling site or not: Cloudflare took their money for years, failed to communicate any problems, then deleted their data when they didn't accept their "enterprise deal". There's nothing saying that they won't do the same to ANY of their other thousands of customers, many of who reads this forum...
Jup, that’s my takeaway. Even if they were in the “right” to stop serving the customer, the way they went about that is absolutely ridiculous.
To (1) - if this was the case, it would have been great if they had talked about it openly or in any way really. To (2) - I do agree that $300 is probably cheap. But I also think that $10k is very expensive, and it seems Fastly agrees.
(3) Mh, I don't think this is doxxing and didn't expect having names would be a big problem. I've just updated the screenshots anyways and censored the names of the representatives.
Cloudflare of course chooses who they want to do business with, but they also pride themselves in being neutral.
Cloudflare certainly handled this poorly in their execution and abysmal level of transparency, but they’re almost certainly purging loss-leading risky customers like OP and they really don’t owe OP the time of day.
OP is lucky CloudFlare even gave them 24 hours. I’m not going to dig through the their TOS or anything but I’m going to guess that you need to have an Enterprise contract to be a business of certain categories like banks/crypto, pornography, and gambling, which explains why they were being connected with a sales team.
OP mentions lost customer trust…but Cloudflare doesn’t want or need OP to trust them. $250 a month isn’t enough to deal with a business like that.
OP isn't the only customer whose trust they lose by handling the issue in this way. It's fine if they want to terminate a relationship with an unprofitable or risky customer, but doing it with insufficient notice to make other arrangements is pretty extreme. In a case of blatant abuse, that might be reasonable, but that doesn't seem to be the case here the way OP tells it.
I did quickly search the TOS for the word "gambling" and did not find it.
13 replies →
The Cloudflare CEO & co-founder is quite active on Twitter[0] and somewhat active on HN[1], would be interesting to get his perspective on this.
[0] https://x.com/eastdakota/
[1] https://news.ycombinator.com/user?id=eastdakota
I don't know how those situations work though. The customer is obviously allowed to say whatever they want, but if a Cloudflare employee or CEO disagreed, would they be allowed to provide their own version of the facts? Wouldn't that go against privacy rules in some way, showing the details of someone's account? I would think they would only open themselves up to legal trouble.
As far as I can see, the author was careful to redact their domain from all screenshots.
I flagged it for him, we'll see if he replies. https://x.com/ArtemR/status/1795074047539068991
- "This also means that if a country DNS-blocks our main domain, a secondary domain may still be available. This could arguably be seen as a violation of the Cloudflare TOS, as they wrote above."
Attorneys love it when people put everything in writing like this.
Devil's advocate:
If a country A decides to block twitter.com but forgets to ban x.com which remains available ... is Twitter engaging in illegality / violation of CDN terms of service?
Like most things in the legal system, it depends on intent. It's pretty obvious that twitter's rebrand to x.com was an actual rebrand and not some way of evading domain bans.
2 replies →
This has always been my concern about establishing a presence online. I've considered blogging about my experiences at work or the cool stuff that I've built and it feels impossible for me to know when I've crossed a legal boundary. How do I know for sure if I'm sharing proprietary stuff or confidential stuff. The lines of legality seem to get blurry real quick.
This is legally equivalent to "we have domain aliases". Lots of sites have those. Do you think that's really the problem here?
No, it’s technically equivalent. I be legally equivalent it would be required to prove (or disprove) intent.
Once you upgrade to Enterprise it's a nonstop 6 month cycle of asking you to pay more.
Couple this with the fact you have a new rep every 6 months and you get some pretty annoying nag service for the entire duration of your contract.
every 1-2 months
From personal experience I know that 10TB per month is like 30k/year and SSL for SaaS is around 40k/year on the enterprise plan. No idea about pricing for having your own IP.
I have no idea why Cloudflare would ask you to use these two features. SSL for SaaS is only useful if you want to add domains and certificates via API.
I have had my fair share of negative experience with Cloudflare but this is next level bad. Unfortunately companies can chose who they want to do business with but it shouldn't be like this.
From personal experience I know that 10TB per month is like 30k/year
What the hell? That's way more than AWS costs, 90% of which would be egress fees. And cloudflare has done a lot of marketing to rightfully call out those egress fees as far too high.
It's the whole enterprise plan, you can't only buy traffic. So you also get all the features which you don't need or want as you can see from the screenshota shared in the original post.
Even on the enterprise plan they don't really start to talk to you about traffic until you are like 3x over your contracted traffic for a couple of months.
It sucks, it feels like they are competing against themselves because they don't have clear pricing or limits.
> 10TB per month is like 30k/year
That can’t be right. I’ve hit 10+ TB within a few weeks on free tier and everything was fine
> Unfortunately companies can chose who they want to do business with but it shouldn't be like this.
If you have a contract with them then they can't arbitrarily choose who they do business with. OP would presumably have a chance at a lawsuit against cloudflare here, the success of which would depend on how well cloudflare argued the ToS violation. A lawsuit might not be worth pursuing here, but this isn't a case of "it can't be helped".
You signed the ToS, article 4 gives them the ability to terminate the service.
1 reply →
> From personal experience I know that 10TB per month is like 30k/year
Is this true? We are at 3TB and growing so I'm slightly concerned
It's not even 100Mbps sustained. That is nowhere near 30k/year or you're getting ripped off.
For 3k/month you can get a good quality 10Gbps link. That's 3.2PB with a P.
1 reply →
If you are doing any serious business you should have a plan B always ready (ie: fastly) and a downgraded state where you can operate without a CDN.
2 replies →
Start making a backup plan
I've done 7TB/mo for many months in a row on the free tier, no issues so far.
Lmao. non cdn traffic for 1tb from an eu provider is 1 euro/tb. And you're telling me 80tb is worth. 30k/month? That's a joke, right?
This is my first post on Hacker News as I primarily just browse. This situation kept me intrigued, wanting to know how it would unfold.
The Google Cloud situation and all these little happenings, including the proliferation of Gen AI into everything, make me long for the days when companies had their mainframes onsite, in closets or separate rooms, away from CDNs and cloud networks. It seems like a better idea to use these cloud networks as a separate off-site backup rather than for primary use.
I’d love to learn more about what will happen next in this saga. I’ve seen a post where a Cloudflare exec has posted here on HN before. They probably won’t say anything for legal reasons, but what repercussions can Cloudflare expect for this? Will they be, or can they be, sued for this downtime and the related expenses?
Did you ever notice the bit in EULAs that states that maximum liability to the vendor is capped at what you paid?
When big cloud goes down, you get a few days of credit. That's it.
When you do custom enterprise agreements you can get a lot more than that.
Unfortunately it's difficult and expensive to scale those traditional solutions to the modern world of billions of internet users located all over the world. It's still quite slow to access a server on the other side of the globe. It's less noticeable for an american user accessing an american company's resources.
Does it matter for 90% of business? I will say no, the vast majority of business is rather localized for very good reasons (as a starter, language/culture/legalese) so it is an unnecessary "feature" most of the time. The reality is that manager drank the Kool-Aid and wants to pretend that they are Google/Facebook or the likes but almost no one else has this kind of scale. Outside of tech behemoth nobody need that cloud bullshit.
> The Google Cloud situation
What's the Google Cloud situation?
They accidentally deleted a customer's account somewhat recently [1][2][3].
[1] https://news.ycombinator.com/item?id=40466810
This doesn't pass the sniff test. From the actions Cloudflare took and their communication, it's very clear that there was something about the way their services were used that they were unhappy about. The post doesn't include what that problem was, but I have a very hard time believing that the author was not in the know and just got their account nuked without any further commentary, especially after being in a number of calls with real human beings from Cloudflare. Surely they'd have plenty of room to both ask and tell to figure out what the issue is. More than anything, this sounds like they knowingly did something shady and are now trying to shift the blame.
So CF is ok with shady for 10k but not for .25k?
Isn’t that a tale as old as the internet? Hosting adult traffic is painful in all respects, and they get a premium for it.
CF was more than happy to help, but .25k wasn’t a number that solved all the problems.
Maybe... Same ratio: I might be ok to do shady stuff for 100k a month, but not 2.5k a month
[dead]
The OP does not give their company nor domain name. I wonder if this is related to recent efforts by the Dutch to collaborate with Cloudflare to prevent online gambling companies operating in the Netherlands.
https://igamingbusiness.com/legal-compliance/legal/cloudflar...
We do already fully block the Netherlands since a long time ago since their regulations don't allow us to operate there.
The KSA license doesn't sound too unreasonable on the surface. What was the hang-up?
I have been hearing stories from developers/entrepeneurs about Cloudflare being very weird to deal with:
"We'd like to talk to you about an enterprise plan."
"No thanks, I'm fine with the free plan."
"Based on your traffic, we'd like to talk to you about an enteprrise plan."
"Is there a traffic limit on the free plan?"
"No, there is no limit. But based on your traffic, we require you to get an enteprirse plan."
[Gives up and gets an enterprise plan]
[6 months later]
"Based on your traffic, we'd like to talk to you about up'ing your enteprise plan to a new monthly pay."
"Is there a cap to traffic in our current plan? I don't see that in our terms."
"No, there is no cap to traffic in cloudflare plans, but based on your traffic, we're going to require you to pay more per month than you are currently paying."
"OK, can you tell me the traffic limit in our current or new plan? So I know what I'm paying for and when I'm approaching it?"
"No. But you need to pay more."
[Wash, rinse, repeat, every 6-12 months]
It seems like while cloudflare technically does not charge for egress, in fact for large egress it's just a game of chicken between the customer and a salesperson every 6-12 months, with the salesperson trying to figure out the most they can manage to get without losing the customer? I mean, I guess that is standard for enteprise sales, but I think usually you at least have some terms to know what you've got for how long without a renegotiation?
You forget to mention that the DDoS traffic causing these issues are also behind cloudflare, but they don't give a damn about them, for obvious business reasons.
Cloudflare controls supply and demand, which, by definition of the law, should be classified as extortion.
> should be classified as extortion.
It meets the definition of a RICO "enterprise". The question is, will anyone bring it up for judicial review?
3 replies →
I thought about using CF in some of my deployments.
After hearing about these sorts of "discussions" from other colleagues, I certainly talked about using their services.
And then I realized that I had to hand them over my DNS? Uhh, no. It could have been "set nameserver to ours in your DNS console".
And also there was the recent SSL spoofing they're doing even with DNS with no hosted websites. And they charge money to send a revocation.
The whole thing is a hot yipes!
>And then I realized that I had to hand them over my DNS? Uhh, no. It could have been "set nameserver to ours in your DNS console".
>And also there was the recent SSL spoofing they're doing even with DNS with no hosted websites. And they charge money to send a revocation.
What's your threat model here? That cloudflare will go rogue and... MITM your users? Can't they do that even if they're not in charge of your DNS? Even if you point an A record to them, that's enough to get a certificate via an ACME http-01 challenge[1].
[1] https://letsencrypt.org/docs/challenge-types/#http-01-challe...
You don't have to. In fact there are some TLDs that they don't even support.
You just need to configure the nameservers and that's it. That's how I do it for mine.
> It could have been "set nameserver to ours in your DNS console".
... that's how it works? They give you the nameservers to use, you set your domains up with them.
You can register a domain through them, but don't have to.
In fairness regarding this particular post, the author admits they were probably violating Cloudflare's ToS, and they knew it.
The folks at CF could have been less obtuse in handling the matter. But at the end of the day this is an online casino breaking ToS and they got spanked.
3 replies →
We had a similar experience. Stuff suddenly started breaking for 10% of our traffic, support dragged their feet for weeks wrt any sort of insight as to what was going on, and then the answer was “you’re over an undocumented limit, try the enterprise plan”.
Fwiw this was some years ago and we moved most of our stuff away from them in response. I didn't get the feeling that this was malicious from their side, more like growing pains / mediocre support people / etc. But the end result was the same as you describe, except we chose not to pay up.
EDIT: more context: I shared this story on HN once before, jgrahamc responded with “please email me”, we did but it didn't move the needle. This further convinced me that CF just has a lot of stuff going on and something weird about our traffic made them error out. My suspicion is that the enterprise plan was supposed to make it internally defensible to pour more engineering resources into our case, but they were never explicit about that which made us worry enough to not do it.
I think that a large reputable business like CF should be clearer about stuff like this. That said, as someone running an API business, I also hold some sympathy for “customer does something weird an unexpected, it’s hitting a limit we didn't even know we had, srsly now what?”. The answer to that should be “work together with the customer to get to the bottom of things, customer might need to make changes too”. They didn't do that, which disappointed us, but I can relate to the situation nonetheless.
We’re still a CF customer, just not for this part of our offering.
> undocumented limit
this makes it sound like the limit is automatic or applies non-discriminately to customers, but my first instinct is that this was manually set by someone, maybe the sales reps again?
5 replies →
Our experience has been quite the opposite once we were forced to migrate from a free plan (a long time ago after what felt like abusing the free plan due to the amount of bandwidth we were using).
The bandwidth caps and all included features were clearly spelled out in the entetprise contract and when we went over, they didn't push for a contract renegotiation unless the overage lasted like 3+ months. And we frequently got new features included in for free.
In fact, recently they asked to renegotiate the contract due to some obsolescence and we ended up significantly dropping the bill as a result. Kind of backfired on them, I wonder if the account manager is kicking herself for this.
It's good to have an alternate experience shared, thanks!
Perhaps the stories I have heard are from people with particularly bad/aggressive sales reps, or who are particularly bad negotiators on their side.
I will say, though, that the free plan is marketted as without traffic/bandwidth limits, and has no traffic limits in it's terms of service, no? If it is possible to abuse it with an amount of bandwidth, rather than this being a "feeling", wouldn't it be more clear and transparent and respectful to just make it clear in the terms?
1 reply →
> In fact, recently they asked to renegotiate the contract due to some obsolescence and we ended up significantly dropping the bill as a result. Kind of backfired on them, I wonder if the account manager is kicking herself for this.
Only if the cost of supporting the depreciated feature was less than the delta.
1 reply →
I wonder why they don’t just have clear limits - seems like it would make it easier to grok.
I think it being difficult to grok is the point, if they laid down exactly how much they want you to pay for bandwidth then it would be easy to go price shopping between them and the competition. But when it's "free" bandwidth, with a fuzzy line where it stops being free, and ambiguous pricing when it does, they can hook people in with a great deal and try to shake them down later.
I still encounter people who refuse to believe that CF bandwidth isn't really free, when you can easily demonstrate that it's not by just observing who uses them. If their bandwidth truly was free and unlimited with no catch whatsoever then every bandwidth giant like Imgur would use CF, but they don't. Imgur uses Fastly, probably because it's cheaper than CFs "free".
4 replies →
Presumably it gives them a lot more flexibility in deciding who has to pay more.
With published thresholds they’re less able to upsell someone just shy of the limit without publicly changing the tiers. Doing that has the potential to upset existing customers who are over the new limit all at once, while also providing intel to competitors looking to undercut them.
They don't want you to know where you are. It's like Kafka but the liberterian edition.
That's not how the "free until it's not" pricing model works :P
IMHO it's just the price finding model that CF has adopted, I expect in the future they'll release limit numbers... unless they decide not releasing numbers is more profitable (i.e. the used car sales pricing model)
Genuine question: Why did you use `grok` in that context?
9 replies →
I wouldn't trust a company after they pulled this stunt just once. Why are you letting them do this to you(r company)?
it's not me -- which is why i'm not nervous about talking about it on HN. I work in the non-profit sector and don't currently use Cloudflare. Just stories I've heard from others.
I'd guess that the cost of switching/cost compared to other alternatives/cost compared to business value/revenue, remained sustainable for the customer, who didn't want to deal with a switch.
In truth, this is kind of how "enterprise sales" has always worked? The salesperson trying to figure out the biggest price that won't lose the customer? But additionally having unclear terms and unclear length of contract (or really no contract locking in your terms/payment) is definitely in the vendor's favor...
1 reply →
Do they offer tangible benefits to justify the higher fees?
That's the thing that gets me about all types of subscriptions / pay walls. You have my attention momentarially, make your best pitch as to why paying you is in my interest.
Sounds like auth0…
But they have explicit limits?
https://auth0.com/pricing
1 reply →
Racketeering is easier and more profitable than actual services.
Shocking how often "gatekeepers" fall to the temptation.
Speaking of racketeering, it's an enlightening experience to search for "stresser" or "booter" providers (euphemisms for DDoS-as-a-Service) and look up their NS records to see who helping them ward off competitors DDoS attacks and keep their origin servers hidden. 9 times out of 10 it's Cloudflare, with the few exceptions being DDoS-Guard, who more or less specialize in facilitating crime.
I wouldn't be the least bit surprised if DDOS protection providers were using DDOS on their prospective customers via proxy. Problem, reaction, solution. Did you pay your "protection" money this month, Luigi?
See also early "anti" virus industry.
This is a really important lesson here. Don't put your eggs in one basket, and if network delivery/etc. is core to your revenues and livelihood, don't trust a random third party host to look out for you.
10TB/80TB at 120k/yr, either way, Cloudflare is taking you for a ride.
If you aren't self hosting, you're really doing it wrong.
What is "don't put your eggs in one basket" here? Your DNS has to point to something... and changing it will go through propagation delays, during which it will be down if you are banned suddenly.
It's not like you can have your domain/DNS somewhere else and point to Cloudflare IPs (to not put DNS and CDN in same basket). Cloudflare does not allow that setup.
You can't protect your website from your DNS provider or hosting provider suddenly kicking you off. You are going to be offline for a couple of days.
> You can't protect your website from your DNS provider or hosting provider suddenly kicking you off. You are going to be offline for a couple of days.
Sure you can. Colocate in two or three places. You're your own DNS provider and your own hosting provider. If one of your colocation companies doesn't like what you're doing for whatever reason, you use the other two until you replace that provider.
2 replies →
10TB/month is 30 megabits/sec on average. Not much for a CDN. They probably need Cloudflare more for DDOS protection than anything else, I'd think?
> Don't put your eggs in one basket
Yet everyone in "ai ai ai ai" is buying up Nvidia/CUDA like there is no tomorrow and then pooping on AMD for even trying to do anything.
History loves to repeat itself.
After reading this article carefully I have a few thoughts. Firstly you were knowingly in violation of TOS after they pointed it out to you. Violating their TOS is a fast way to have your account suspended.
Secondly I'm a little confused why they would require you to pay a year upfront? I would like to hear from cloudflare as to why they required this? It's pretty fair for them to ask you to pay a year in advance because of the risk that you carry as a gambling company.
Cloudflare needed you to have to enterprise plan to remove liability from them. It's not even a big request, they have specific pricing plans for a reason.
Every customer is in violation of the tos if you read the html-only section
They know this
There is a single mention of html in their terms, and it just restricts wrapping cloudflares software. Could you link me the section where they restrict anything to html-only?
Why is HN demoting this article?
It was at the top of HN, then quickly buried to #20-#30. It is now at #27, being a hour old with 318 points.
It was significantly downranked after about an hour on #1: https://hnrankings.info/40481808/
Yep, from #1 to bottom of the first page in 5 minutes...
Looks like a group of people decided to bury it
It is actually more likely it is the opposite, that it got temporarily boosted to get exposure, and then fell back as interest vaned. If mods want it gone, then it will be gone.
175 comments and +350 points in 1 hour is anything but "vanished interest" imho.
In the last 10 minutes, it got 35 more upvotes, but dropped from 27 to 30.
It has become apparent that doing business with Cloudflare is a liability.
This is the nail on the coffin, but make no mistake, Cloudflare has been a liability. It's a massive Man In the Middle decrypting all traffic, including OkCupid and 4chan for example. Imagine all those 4channers learning they aren't actually anon.
That's literally their business and why people use Cloudflare.
Caching, detecting+modifying headers, routing traffic, ...
The recommendations to, basically, not keep all your eggs in one basket and have backups of config are surely good ideas. But if you have to plan on dropping cloudflare in some arbitrary 24 hour window, perhaps it’s CF that’s the problem. This sort of stuff and other recent articles about CF are so worrying that it’s now being run by the finance team (hence why every email they got in this article was from sales teams rather than any technical folks).
Also; if not registering domains on CF does anyone else do at-cost or otherwise super cheap pricing?
I’d say be wary of any public company and actively make plans to get out of any company that gets acquired by private equity.
Actually had a sales call with Cloudflare in the last month and I got some bad vibes from the whole experience. We did not end up going with them.
May I ask who you ended up going with? We had a similar experience recently and have some concerns with anything on CF.
1 reply →
Wisdom of the ancients: Always make sure your domain services are completely independent from your other service provider(s), regardless of whether Cloudflare is involved. (Sorry, no recommendations at this time.)
Do we know of any alternative services providing at least the basics of what Cloudflare does?
Such as:
CloudFlare does not provide unmetered anything, at best they provide services on a discretionary basis while trying as hard as possible to make it appear this is not the case. It's better to think of their product line as a CRM system with some CDN features on the side
To be fair to CloudFlare - let's replace "unmetered" with "fair use".
Very typically free = actually "fair use".
Where it gets murky is when this becomes a shotgun sales tactic.
Easy to build in a few weekends, grab a few geodistributed racks with Mellanox NICs and do HW offload. Obj storage is a similar NVME DMA approach.
How is this not still #1? 488 upvotes in 3 hours. It was number one, the right to the third page? sus...
This reads more like a shakedown than anything. Even if the casino was being dodgy, CF went in asking for more money, not demanding that they stop doing whatever it is they were doing.
This sort of situation is not time for an angry blog post. It's time for lawyers. OP needs to speak with counsel and find out whether they have a claim against Cloudflare for interfering with their business in this way. (If OP's business is so illegal they can't get a lawyer to help with that, that's another story, but it sure doesn't look that way.)
Fundamentally, the OP might be involved in something scummy or at least against Cloudflare's TOS. But if that's the case, if you have a customer who is violating your TOS, you don't hit them up and say "pay me an extra $119k a year and I'll look the other way". You say "here are your violations, fix them and prove to me that you fixed them, or pay for plan X which has terms under which they are not violations."
The way Cloudflare handled this is completely inappropriate and even if it wasn't their intention, makes it seem very strongly like extortion. Two wrongs don't make a right, and OP's business being possibly shady does not give Cloudflare license to extort them.
What are lawyers going to do about the clause that says they can drop anyone any time?
Trying enforce "reasonable behavior" by suing is a massive money pit that might yield nothing at all.
Doesn't matter if you're an asshole company or not: always have an exit plan and test it.
I’ll be honest, the high pressure to pay almost seemed like a well devised scam or phishing email.
Scammer does recon on victim. Notices they use CF. Use high pressure sales tactic to get them to pay a hefty sum up front or else lose access.
But as you read on, I see company did their own DD and followed up directly with CF executives and teams. Confirmed account is locked at CF.
In this case, CF is acting scammy.
I wonder if they are having liquidity issues thus the push for high pressure sales tactics and blackmail.
Way to bury the lede of OP running a gambling site and doing a ton of shenanigans to make it legal in different jurisdictions. I understand why you might think you shouldn't lead with that but it puts the entire response from Cloudflare into perspective as dealing with gambling sites sounds like a headache and it's reasonable they might not want to run that kind of venture on a regular plan.
Rule #1 in any work, business, or even personal and social life: never put all your eggs in one basket. For CF first example, only use the cdn part, don’t use the registrar one, there are more registrars out there, don’t use their worker, use something else, and do on. It’s for obvious reasons, when shit happens, you can mitigate the impact quickly and easily compared to using all at once.
They figured out that they are losing money with you and offered to negotiate a new plan which was declined so they made and offer which was ignored and you let them know that you maybe want to change the service so they magically found an violation and after still not accepting their offer they took you down.
Pretty standard behavoir from both sides with room for improvement. They should have been more clear about what's going on and you should have been more insightful what they wanted.
In my opinion they acted to fast and not really cooperative, but if you wouldn't have declined the initial offer and started to figure out why they offer it and what options there are, you would have came out with a better deal than 10k/month and likly without 1 year upfront payment which would have given you the time needed to transition to another service.
Meta:
455 points, 3hrs old, but on the 2nd page of HN. What's up with the algo?
I'm not sure. It was on spot one on the first page, then something happened and it got downranked: https://hnrankings.info/40481808/ maybe due to being flagged by people (according to another comment).
I guess it's due to general negative sentiment towards casinos, which may be understandable but doesn't (in my biased opinion) change anything about CF's behaviour in this post. I would have left it out but it's necessary in order to provide the full context.
Is this why cloudflare manages to be cheap?
Their business model is described in their S1 filing: https://www.sec.gov/Archives/edgar/data/1477333/000119312519...
* Cloudflare operates at a scale where its caching saves a lot of bandwidth, which saves ISPs money, which makes Cloudflare an attractive partner for peering and co-location.
* CDN is a platform on top of which Cloudflare can offer a lot of additional services that used to be expensive dedicated middleboxes.
They manage to be cheap because it's massively multitenant infra
It's unclear as it shouldn't be possible to be cheap. That said, in a world of data, Cloudflare being a massive man in the middle (MITM) probably means something [1].
[1] Cloudflare decrypts your traffic, reads it, and then forwards it. They see all encrypted data going to and from your website, in plaintext.
This data collection project has a name: "Project Honey Pot" (and had excellent relations with FBI), look closely the story that led to Cloudflare.
https://www.projecthoneypot.org/cloudflare_beta.html
2 replies →
Or CF had already decided to kick them off the platform and tried to get some money before they did so.
The casino in question is Gamdom btw
Im reading between the lines here but it seems like the traffic amount, the saas subscription tier, and the actions required to remidiate some issue were all unaligned.
1. Its quite possible thar CF having this site on some multi-tenant infrastructure could be threatening. Not unreasonable at least to ask them to have their own IP block.
2. If thats the issue then a clear explanation should have been provided. Routing to sales is inexcusable. Someone isnt being transparent.
3. If it’s a pure cost / revenue issue then say that, set a deadline and negotiate. This is bad karma and even though CF is clearly the market leader, what they do isnt rocket surgery. Not worth it.
My thoughts here are also all speculation, but when you mentioned multi-tenant issues my mind immediately went to a situation I've seen all too many times before:
- a companies ops team identifies a tenant that is too heavy/burdensome for multi-tenant infra and is causing issues. These issues can cost a serious amount of money if you factor in dev/ops times to resolve, other customers impacted, etc. Certainly more than what a hypothetical single multi-tenant customer could be paying
- they escalated internally and need the tenant moved to enterprise asap to resolve
- the only reason the tenant was on multi was because sales sold them the wrong thing, so now it's on sales to explain how to fix this
- improper handling internally results in this landing only on sales, with no backup, and with their task being to get them to take enterprise
- when the customer refuses enterprise they go "we've tried nothing and we're all out of ideas"
Again, this is totally speculation and I'd hope CF has more mature practices than this but this is a scenario I've seen before in much smaller orgs.
What stands out as odd to me is that CF seems to be pushing away a $10k/month customer. No business can reasonably be expected to accept sudden price changes like that, even if they'd paid, they would've moved away within a year.
Given that the article is an online casino that seems to be using potentially ToS violating domain rotation, and that they pay so little per month for apparently millions of users, I for one will not form an opinion on CF based on this article before CF has a chance to defend itself.
If you depend on one vendor, as CTO always have a plan-b prepared that you can pull out and execute. Stall, stall, stall while you're executing your plan.
$120k will never be enough, price hike is incoming for renewal.
Okay, with this thread, I'm learning and need to:
One Question: For the Web site for my startup, I have the ASP.NET code running so ASAP will be getting into to a business account with my ISP, IPs, domain names, DNS, etc., at least for the Alpha Test.
So far, my intention is to host my own Web server. I've heard of CloudFlare, how they can help stop DDOS attacks, etc. but so far have hope not to use them.
Question: How realistic is it for me just to host my own Web server and, e.g., avoid any chances of problems with CloudFlare, the Cloud, VPNs, etc.?
Thanks!
Regardless of who was in the right....it scares me how much cloudflare and the cloud have become ingrained in the internet.
Imagine getting banned by cloudflare or some other cloud provider....
has this happened to any businesses that were not questionably legal?
There are links in the article, for example:
"Small SaaS banned by Cloudflare after 4 years of being paying customer"
https://news.ycombinator.com/item?id=31336515
In both examples you provided it was less "banned" and more "switch to a higher priced plan or we'll kick you off". In both cases they seem to be bandwidth related, and in the second case specifically they mentioned having hundreds of terabytes bandwidth but were upset for being told to upgrade to the $200 plan.
Thanks, I was genuinely curious
> causing… irreparable loss in customer trust
> I'm a SysOps engineer at a fairly large online casino
Oh no, a casino losing the trust of its customers? Those places are normally so scrupulous!
Billion dollar unregulated casino that is burning through cloudflare ips.. i don't see where cloudflare can be blamed here, they stated you should BYOIP, you didn't approve and were banned, their is no blame here.
Just FYI some countries ban casino domains/ips that are not licensed to operate even when its "just a landing page that says sorry not available"
Can a sales rep de-platform you?
I hope that’s not the case, because that would allow for bad behavior by reps trying to manufacture end-of-quarter sales.
EDIT: why the down votes?
I moved away from Cloudflare—to self hosting our network infrastructure—because, while this didn’t happen to us, I was very aware that it could. We had a great deal on Enterprise for a couple of years, but zero guarantees that it would last (and some indications that it wouldn’t). I wanted to stop praying that they wouldn’t alter the deal.
Sounds like the famous Oracle licensing team.
> When we told them we were also in talks with Fastly, they suddenly "purged" all our domains
Holy shit.
What a way to more quickly show your customer the door.
What’s especially shocking is how closely coupled sales and engineering are. Like I get they talk, but for a sales call to end in engineering pulling the plug…
To me it's pretty clear: Trust & Safety (NOT engineering; they don't appear to be involved) likely raised an alarm saying "Customer X is breaking TOS - no immediate resolution available, but something might be possible given extensive legal & engineering review. Recommend switching to enterprise so we can study how to make it work."
In that light you can see why Sales would be sent as the messenger. But I agree they shouldn't have been involved. Sending a T&S representative would have been better. But then again it looks like from screenshot #2 that they kind of did that? They just didn't have a direct call with T&S.
Nothing shocking here as someone who has been involved in hosting for close to twenty years.
Another issue with Cloudflare is that it seems to be blocking VPN access to sites that have any regional restrictions.
Of course, it is easy to identify the IP addresses of the well-known VPNs, so it's not rocket science, but it does mean that popular VPNs will no-longer give you out-of-region access.
Notable, their Enterprise plan quote included BYOIP. I think that's the kicker. Cloudflare likely got a few of their anycast load balancing IPs blocked in one country, causing a huge disruption, because this customer that makes them no money wasn't in full compliance with local laws.
If that is the “key” they should be transparent and explicit about it. Now it seems CF is a Mafia that realizes one of their extorted business should be squeezed to death for more cash.
Casinos should indeed be squeezed to death, and the money should go back to their victims
> I'm a SysOps engineer at a fairly large online casino.
And for some reason Cloudflare's the bad guy.
For some reason both cannot be the bad guy?
DDoS services are protected by CloudFlare because free speech and legal online casino are bad guys. Good.
$250/month sounds like nothing at all for a site with a claimed 4M MAU. The enterprise rate of $10k/month sounds a lot more reasonable. If everything presented here is accurate, I'm not understanding the sharp discrepancy in pricing tiers. If anything they should've already been paying more than $10k/month for massive traffic on the basic plan and then be able to save money by paying for massive scale when negotiating rates for the enterprise plan.
Also this sounds like an online gambling site of questionable legality, knowingly serving customers in jurisdictions where it's illegal, so I can't say I have too much sympathy, and I feel like Cloudflare effectively fired them as a customer when they realized what they were up to.
According to Cloudflare themselves, 80TB traffic should cost around $100/month. See: https://blog.cloudflare.com/aws-egregious-egress
That's just one of many components of what you pay them in total, though.
1 reply →
The amount shouldn't matter, it's the unprofessional response from CF that's of concern. Also, the author doesn't necessarily say that they would be unwilling to pay more or even negotiate a higher price. The author has made it explicit that CF were more or less unwilling for any practical conversation. That, to me, is the problem. A lack of professional courtesy, communication, and transparency. Obviously there may be details from this exchange which have been omitted but if I were in this position, I would be equally upset.
So when they asked to pay the 10k monthly, was that to gain time to move or was the price acceptable? Does it say anywhere?
Every business with this size should have another CDN as failover, relying on a single provider is proven to be dangerous.
I'm surprised by how many comments seem to assume Cloudflare is at fault. Shouldn't the default assumption be that no one did anything wrong?
In defense of Cloudflare, the sys ops engineer should have understood the situation and knew they were misusing Cloudflares services. They decided to play hard ball by bringing up the fact they were thinking of leaving. And we have no history of the multiple phone calls they had with Cloudflare.
Should have paid the protection money to CF. They need their cut of your gambling bag!
So that's how you make the internet a safer place. Good job Cloudflare!
While the author could improve the narrative in his article, the historical issues with Cloudflare combined with, yet another one, paint a stark picture.
Combine it with the stories I hear about Sales, the numerous other PR fumbles already mentioned in this thread, and the months I’ve personally waited (while on a paid plan!) for ticket responses only to get cookie cutter responses is, quite frankly, embarrassing.
CloudFlare puts in a good front, and their products seem decent, but they really have questionable business practices that should make anyone think twice before using them.
Naively, I imagine that Cloudflare’s math looks something like this:
(Amount owed by customer at end of month times the probability of on time full payment) minus Cost of providing service to customer for one month = profit
Since this is an online casino, could the risk of late/under/no payment be quite high?
Beyond Fastly, what are viable non-extortionist alternatives to Cloudflare?
Huh. Now that was some high-stakes poker right there. It seems the casino knew they were breaking the TOS and paying too little and Cloudflare caught up with that. Then knowing their situation they decided to ask for payment for all the expenses of the previous years (and some extra). In quite passive-aggressive manner.
But the casino still decided to stretch the penny and alas, whoever at Cloudflare was in charge got quite upset their extortion-tactic failed. So they decided to resolve it the American way and kick them out with zero warning - ouch! How fascinating.
I myself like using Cloudflare as it's quite affordable to setup and use. Makes me sad to know they have to resolve to tactics like this to finance their service. Well, at least I don't work in dubious businesses that violate TOS so perhaps I can at least wish for a graceful termination when my Enterprise bill is due.
I guess it's the natural cycle of money always spreading its tentacles to everywhere, and specifically applying pressure after sufficient metastasis and entrenching.
> Make backups of your configuration on Cloudflare. It's an unexpectedly large pain to recreate all those configurations
Better yet, configure CloudFlare through terraform, so all your config exists in your own repo all the time. It also helps day to day since it's not that hard to accidentally flip some switch in the dashboard.
But yeah, do research alternatives. CF has too much power already and will either ignore issues, destroy you, or pay lawyers to protect people trying to get you murdered, depending on their mood. There are better options.
Please stop using cloudflare, cloudflare captcha and google captcha is spyware and it needs to go away.
> captc
This post was definitely demoted by HN. It stayed in the first position for less than 5 minutes and, as it quickly gathered upvotes, it jumped straight into 24th and quickly fell off the first page as it got 200 or so more points in less than an hour.
I'm 80% confident HN tried to hide this link. It's the fastest downhill I've noticed on here, and I've been lurking and commenting for longer than 10 years.
I had to search for it. I was under the impression that HN removed it for some reason.
Does HN has stake in cloudflare?!
Ranking is strongly impacted by the flamewar-detector. Affected threads are automatically downranked to cool things down until a moderator steps in and manually reviews it.
Page 3 at #70 after four hours.
yep, it happens every single time a negative story about cloudflare appears
more than a coincidence
That’s why I use hckrnews-com as HN front-end.
> I'm a SysOps engineer at a fairly large online casino. We have around 4 million monthly active users. We had been happy Cloudflare customers since 2018 on the "Business" plan which has some neat features and costs $250/month for "unlimited" traffic.
Sorry to be “that guy,” but, you’re serving 4 million people at a casino and paying $250 a month for shared multi tenant infra, and you’re SURPRISED you have problems? Really?
To be honest, I’m glad these sorts of businesses get kicked off Cloudflare because it causes problems for others sharing the same IP space and infra. I’ll let someone else with experience discuss how many times a day the network would see a hacking or DDoS attempt against the online casino, which is by far the favorite target of hackers. But in general, I just don’t want any of my infra touching the same stuff as these guys.
Like another person here, I am assuming that Cloudflare ops told someone “tell these guys to get their own IPs and upgrade,” and then the message went to Cloudflare’s (utterly lousy!!!) sales people to try to fix before shutdown, and then it all turned into the mess we see here.
The true moral of the story, I think, is, if you’re running an online casino on a shoestring budget, expect bad things to happen to you. Of all kinds.
What happened to net neutrality? Why bring in sales if the issue is a legal one.
WOW
Just wow. We were in the midst of negotiations with Cloudflare and I’m Going to hard nope after reading this.
I’m guessing they aren’t doing that well and are looking for revenue to cover the holes.
Sales teams can be asinine to deal with. I don't think cloudflare realises how damaging this is for them long term.
Using this as a blatant example of why digital anarchism is needed nowadays
[dead]
[dead]
Isn't there some kind of law against companies extorting the customers and being evasive about their terms of service and their prices?
Customer protection laws usually only work for individuals, not companies.
[flagged]
[flagged]
Well, this is disgusting behavior CF. I wonder if OPs company suing CF?
Indeed. Based on what has been shared by OP, they could have a case.
If OP’s business was in fact illegal, CF should have stated it. Now it seems CF is an evil sales driving monster. A monster that grew so big it thinks it can do whatever it likes.
The sad part is that, assuming OP is not leaving out critical parts, multiple people play parts of this evil machine. I’ve seen how sales people think. But this is next level toxic culture. The second customer threats of leaving for the competition, they freak out and pull a bigger lever to destroy them. And the fact that a company allows this to happen…
I would never do business with CF. Good thing i don’t right now. Cause i will definitely take it elsewhere.
A paragraph ends with:
> This could arguably be seen as a violation of the Cloudflare TOS, as they wrote above.
And the very next paragraph begins with:
> In any case, we receive >95% of our traffic through the main domain that’s been unchanged since our founding, and were happy to resolve this issue in whatever way...
And then they complain about paying up?
The only issue I see here is around the aggressiveness on the CF side. But, I was not in those meetings and the way above reads tells me that I might have been slightly mad so perhaps the CF was just taking it out on them?
Anyway. I don't think this is a CF foul.
>I'm a SysOps engineer at a fairly large online casino.
So they did a good thing taking it down, no?? Addiction as a businessmodel is not that cool
Sidestepping the whole ethical conversation and just taking it as a good action for the hosting provider to do it still fails the sniff test as Cloudflare (according to this story at least) didn't seek to take it down rather sought to make more off of it.
Sure, but if two bad entities fight I don't care about any of them, let them fight to death.
1 reply →
After they deplatformed KiwiFarms, I thought that's an isolated case but turns out they are just unprofessional. I can't have pity for a casino service anyways.
> a web forum that facilitates the discussion and harassment of online figures and communities. Their targets are often subject to organized group trolling and stalking, as well as doxxing and real-life harassment.These actions have tied Kiwi Farms to the suicides of three people targeted by members of the forum.
Sounds like a pretty abhorrent website
CloudFlare only dropped Kiwi Farms because Keffals made it inconvenient for them to do business with enterprise customers. Said customers were looking at Twitter and saying, "Wait, you host WHAT?!"
Before that CF was high and mighty on the "free speech" horse.
There's an old spat between CloudFlare and Malwarebytes where MB was threatening to block all of CloudFlare because they wouldn't remove literal malware. The argument being that running a reverse proxy "isn't hosting", and should be treated differently, even though to literally anyone else there's no difference between an origin server and a proxy.
CloudFlare is just sketchy as all get out, IMO.
Putting my biases on the table: I think CloudFlare shouldn't have hosted Kiwi Farms in it's current state, because I don't think hosting dox should be legal. A website that hosts dox is not engaging in speech, it is engaging in censorship. Hell, in the EU, it's already illegal to host dox, the US just needs to pass a privacy law comparable to that of the GDPR. Kiwi Farms is legal in the US purely for the same reason why the CIA/FBI/NSA can legally buy advertising data from Google and Facebook.
2 replies →
Who wrote that?
Didn’t they also deplatform stormfront and a few others they didn’t like?
Yes but then people argued it's about these websites being bad. Turns out it's just about money, considering Cloudflare has no problem with CP or other website calling for violence and celebrating it.
Cloudflare is evil, even though they do a decent job of pretending they care. They are smart to offer gateway services (gateway as in gateway drugs - stuff to get you hooked) for free to end users, since it grows their fanboi base, gets more people familiar with their services, and gets people hooked in ways that make it very difficult to use something else when their projects grow.
However, any reasonably competent person can see that recentralization of the Internet is a Bad Thing™, and that this is precisely what Cloudflare wants.
Likewise, we know that aggregating our data through a for-profit company that's based in the United States means that collected data is reasonably in the hands of the NSA, which makes their DNS-over-HTTPS scheming suspect.
Just like what happened with the company in this post, we have plenty examples of them abusing their position to extract money from both legitimate companies, like this one which is aware of their legal obligations in various countries, and scammers and spammers alike, who Cloudflare are more than happy to host indefinitely in the name of "free speech".
Their lack of clear communication, their broken abuse reporting, their continued claims that they don't "host" all show them to be antagonistic towards anyone negatively impacted by their facilitation of illegal activity.
Cloudflare is an evil company that just happens to be better (but not great) at hiding it than other evil companies.
The consistent reaction when I brought up Cloudflare with other CDN technical and sales teams, as of about four years ago, was a laugh and something along the lines of “yeah we’ve got some customers with some stories about them”.
Bait-and-switch seemed to be the most common pattern, plus crazy-high prices once you’re on the “switch” side of things.
But their sales team was so uniquely uninterested in our business that I never had to find out first hand.
> However, any reasonably competent person can see that recentralization of the Internet is a Bad Thing™, and that this is precisely what Cloudflare wants.
It's an inevitable outcome, as long as there is nothing done against the big threat actors: government-run APTs from China, Russia, North Korea and Iran, government-tolerated scammers (India, Turkey), rogue actors in our governments' security services (e.g. Pegasus), ordinary criminals mass-hacking vulnerable devices and selling access to them to be abused for DDoS'ing for less than the cost of a coffee at Starbucks... it's a wild west, and people are hiding themselves behind the largest giants they can find: Cloudflare, Akamai, AWS, Azure and GCP.
[1] https://en.wikipedia.org/wiki/Pegasus_(spyware)
You're throwing unrelated facts at the statement. Recentralization has absolutely nothing to do with being protected from DDoS or from other threats. Now, DDoS (and other kinds of) protection can be done by recentralization, but that's just one possible way.
Saying it's inevitable makes you seem like a Cloudflare apologist, which unfortunately we see way too often here on ycombinator. Has anyone refuted my suggestion that Cloudflare is knowingly evil? No. Have they downvoted because my information is incorrect? Also, no, or if they think I'm incorrect, they haven't bothered pointing out how.
People want to like the things they choose, and this, unfortunately, is where Cloudflare is cleverer than other large, evil companies.