Comment by candiddevmike
8 months ago
Sounds like OP is a casino and plays domain games to avoid regulatory interest. Recommend reading article carefully before reacting to the headline. Hopefully Cloudflare provides a perspective.
8 months ago
Sounds like OP is a casino and plays domain games to avoid regulatory interest. Recommend reading article carefully before reacting to the headline. Hopefully Cloudflare provides a perspective.
Hmm. My take is the casino structured its business to comply, not to evade interest. Further, I don't see how Cloudflare benefits by taking on the risk to charge more to help a customer avoid scrutiny. More like: they know it's a humming business and want a piece.
The way I read the screenshots of the emails from the articles seemed to suggest that something the authors company was doing was causing issues with IP reputation on CloudFlares range.
Them very aggressively highlighting the BYO IP feature and then even suggesting third parties to rent IPs from strikes me as a significant detour from their normal “script” (having dealt with their AU sales team before).
CF calls and says there is a problem with traffic. They want to push an enterprise plan. Customer says no.
CF calls and says there is a problem with domains. They want to push an enterprise plan. Customer wants to solve problem, dropping domains, making changes. CF says, only enterprise plan will remedy the situation.
There is obviously a sales script involved.
“get back to Trust & Safety"
Heard that story several times, it's always another team, e.g. "Licensing" that need to be satisfied, or that if you don't pay up, that team will be off the leash. Also heard the pay-for-a-year-upfront for several large vendors who pull this. The reason is, some sales reps need to make numbers, so they shake the tree and see who falls down:
"Cloudflare has absolutely no information on when they will force you into custom billing, but when they start "urgently" needing to talk to you you're probably not going to get out until you have a juicy custom contract with them."
this is exactly what is happening. Cloudflare uses an anycast network, so IPs are shared by default.
this customer is damaging Cloudflare IP reputation which hurts other customers. Cloudflare can either fire the customer to protect other customers using Cloudflare IPs, or force this customer to use their own IPs and damage/manage their own IP reputation.
unfortunately this is expensive and OP is mad they can't do their legally fraught gambling operation on Cloudflare's addresses for free
9 replies →
Compliance:
> We do have multiple domains that mostly act as mirrors to our main domain. We have these for a few reasons. One is that since we are a casino, we have different regulatory requirements we need to comply with in many countries.
Evasion:
> Another is that we use them to target different global user groups and affiliates and track conversions long-term. This also means that if a country DNS-blocks our main domain, a secondary domain may still be available.
This is more like one gang hitting up another for "protection" payments. I had to laugh when they called it "Trust & Safety".
> My take is the casino structured its business to comply, not to evade interest
It's impossible to say what's going on since it's an anonymous post with no details.
Maybe it's all 100% true.
Maybe there are some key details being left out. Wouldn't be the first time I've seen one of those outrage posts that seriously misrepresented things.
Whatever the case, obviously the author is not an unbiased party. These posts do well because "zomg Cloudflare bad!", and maybe they are, but I sure as fuck don't trust some casino guy either.
Thing is, this is also not the first time when Cloudflare T&S team has been disrupting their customers out of the blue. The post even has some links to other HN stories.
But for $10k a month cloudflare is ok with that? Either it's acceptable or it's not, there is no way that this looks good for cloudflare either way.
A reasonable scenario to me seems to be: An automatic "upgrade to the enterprise plan" requirement was triggered, and then in the process of the sales calls to make that happen, Cloudflare got serious eyes on the customer for the first time (whereas at a paltry $250/month previously they wouldn't have), and realized exactly what line of business the customer was involved in, and decided to fire them.
I was rushing to judgment until I heard this... pretty plausible.
In support of your theory particular is I don't think enterprise sales "ragequits" a conversation when the customer is mid-evaluation based simply on the idea that they are considering multiple options.
Why would they walk away at this point, let alone ban the customer.
From the write-up I bet CloudFlare had it as a "60% to close" in their CRM at this moment. It doesn't make sense for them to drop the ban hammer in this moment.
PS: explanation or not, this is deeply shady behaviour from CloudFlare. Just perhaps a little less so.
3 replies →
This actually seems reasonable, and a potential part of the narrative the original poster would be likely to leave out.
1 reply →
If it's legal but burdensome (somehow) to host a particular industry, requiring more money to deal with the increased burden seems reasonable. For instance, if their legal department needs to deal with complaints from various countries, that probably costs more than $250/month.
That being said, I doubt that's the core issue in this case.
That isn't how the world deals with risk.
If you think something your client wants could explode into a liability, you can turn them away or you can just make sure their bill covers your exposure.
If it's a legally questionable service, there's likely to be plenty of abuse contact, or they're going to be a big target of crime, they're going to end up paying more. This is the same reason why some industries (eg porn sites) have always paid more for card processing.
It's not just 10k a month. it's 10k a month for the plan that allows you to BYOIP (Bring your own IP addresses). That was cloudflare's issue.
Their business was causing IP reputation damage and all plans but the enterprise BYOIP plans share the same IP pool.
Essentially it was "use your own IP pool and pay us for the cost of maintaining that pool for you or GTFO".
This wasn't just a normal sales rep hitting them up. This was trust and safety (i.e. the moderation team) coming to them with a compromise that would allow them to stay on the platform. They chose against that and were dragging their feet.
The timeline of the article also really makes this clear. This wasn't over the course of 24 hours. This started a full 4 weeks prior with sustained back and forth. They only included a few images of emails from the discussions but the article makes clear that there was more discussion happening.
And to quote the article. After receiving the ultimatum, they got an entire extra week to deliberate.
> We managed to buy a week of time by letting it escalate to our CEO and CTO and having them talk directly with Cloudflare.
Then finally when they told CF that they were just buying time while looking to move elsewhere, CF dropped their act of goodwill and the moderation team resumed the moderation action they would have taken in the first place had this been a smaller account.
----
So yeah it sounds bad from the snippets but this was basically "hey you are a big customer and you are breaking rules we would normally ban anyone else for but if you can compensate us we'll spend the labor hours and infra to let you keep operating in your own little quarantine box.". So this really should be seen as an act of goodwill rather than malice.
You can't start the timeline from the first email, because clearly Cloudflare didn't communicate the actual issue to the customer. (Yes, the customer could be lying about what was said in that meeting, and they could have been told what the problem was rather than it being just Cloudflare trying to upsell them the enterprise plan without telling why. But then the "omg, we just discovered a problem with your site during a routine inspection!" email sent two weeks later wouldn't make sense.)
They also were clearly lying in those email messages: The second email says that domain rotation is strictly forbidden, but a few days later in the third email they're explicitly selling features for rotating domains more effectively.
And sorry, but a company selling "we'll override the Trust and Safety team if you pay us $$$" is absolutely unacceptable. There are only two options, both bad. Either they're not running a real TnS operation, but just pretend-staff one in order to run these kinds of shakedown operations. Or they're running a real TnS team that found a real problem but are letting sales people override the TnS team's honest judgement.
> So this really should be seen as an act of goodwill rather than malice.
It's called "extortion"
3 replies →
I can reason my way into it, I think objectively. To protect their IP reputation, CF required BYOIP. This costs them something, and de-jure requires an Enterprise plan. Which for the customers usage costs $X. Is it right? Ehhhhhh. Does it follow corporate logic? Yeah. (Sales logic? YES)
I'm not defending Cloudflare's exact actions in this scenario, but it seems reasonable that there are cases where yes, for $10k Cloudflare is okay.
Risk can be mitigated, especially if you take care to know what the risk is, but risk mitigation and the salaries of the risk mitigation teams are not free.
The answer of "no, we will not host you unless you pay us enough money to hire people to make sure we're not breaking laws by hosting you" makes plenty of sense, and an online casino that is likely dubiously legal in many countries is definitely a place where you might use that answer.
I'd also expect there are cases where Cloudflare enter into enterprise agreements with customers, get a good hard look at exactly what's happening, and then tear up the agreement and walk away.
And all of that is fine when communicated properly. Even if OP is an unreliably narrator are we to believe they also left out some of CF's emails?
To me it looks like https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_pr... is entirely the wrong email to send in the situation and if you are as old as I am and come from where I come from, you will have flashbacks to "reading between the lines" of the party daily in the 1980s. The real content is at the bottom:
> As we have a very short window to report back to Trust & Safety team, please let me know if you can make time tomorrow
Big red flashing lights: the right questions are 1) why is T&S involved at all 2) What are their concerns which forces such a hurried deadline? 3) What are the consequences of missing this deadline.
The right email would start with something like this:
> Providing services to your business constitutes serious legal risk to Cloudflare. We are happy to work with you in the future if you are buying an Enterprise plan. As we need to commit significant resources to accommodate you, we need an annual commitment. Otherwise, with much regret we need to terminate our services provided to you as it is our right per Terms on date/time. ("We may at our sole discretion terminate your user account or Suspend or terminate your use or access to the Service at any time, with or without notice for any reason or no reason at all.")
> This plan would also include these features:
1 reply →
That's not true at all. That line of argument gets close to "if this product is free for open source, why is it not free for me? either it costs something to operate or it doesn't." You don't get to price the service.
In this case "the service" would be to look the other way on illegal activities for $10k/mo.
I'm not saying cloudflare can't do it, I'm just saying it's wrong.
The point is more that the author is an unreliable narrator and you need to apply a little salt to the rest of the story. Cloudflare absolutely shouldn't be taking bribes to permit regulatory evasion. But if they are, I want more evidence than a substack post.
It was the opposite? To comply with regulation.
2 replies →
It also seems strange they dont know their Traffic Numbers.
>Note that 80TB is the number they tried to sell us, I don’t know if it is accurate since they removed all our access to historical analytics.
I mean you dont need accurate Data but surely most would know by heart their traffic in rough figures? Or am I the old dog where every new Web Dev are so used to Cloud and Serverless they have no idea what they are using?
7 replies →
Taking a step back, why would they even care if their platform is supposedly neutral and not responsible for the content ?
If they can indeed stop providing services to a casino, why cannot they shutdown a website spreading pro-war propaganda, or a website selling illegal services ?
It means they are making editorial choices, instead of just being the technological provider and being a neutral "internet pipe".
Not sure it's really in their best interest to self-police in the end, as they could lose their DMCA safe harbor provision ?
> Taking a step back, why would they even care if their platform is supposedly neutral and not responsible for the content ?
Because their main network all uses one big IP address pool and the blocks by various regions/countries against their site were probably not just DNS blocks but also IP address blocks.
So they now have an account whose activity is getting their IPs banned in countries where they operate.
So they told the account owners they needed to pay for an enterprise account and a dedicated IP address pool maintained by cloudflare. That's why CF kept talking about BYOIP in the emails.
i.e. "Pay for us to build you a quarantine with your own IP pool or leave ASAP"
This may indeed be the motivation but, at least in those emails that are presented in the linked post, there's no evidence that Cloudflare did at any point clearly communicate that this is indeed what the problem is.
> Not sure it's really in their best interest to self-police in the end, as they could lose their DMCA safe harbor provision ?
This.
That said, we're seeing this across so many platforms, from datacenters to social network sites.
They blew their safe harbor provisions years ago and yet remain untouched despite this.
It’s not “editorial” to choose to stop serving (or charge more money to) a customer whose actions pose legal difficulties or risks to your business. If some country’s vice division contacts cloudflare legal due to a customer’s illegal online gambling presence, I guarantee that cop does not care about a US/EU copyright law.
The same thing is true for IP reputation, just without an external official complaining. If other CF customers are negatively impacted by one customer’s action, CF isn’t violating safe harbor by booting that customer or passing on the costs of mitigating that impact. That’s just running a business, not exercising editorial review of hosted content in violation of safe harbor provisions.
I do encourage you to read the whole article cause there is some fine details in there. The main point is that we were happy to remove any domains apart from our main domain (which gets > 95% of our traffic) but Cloudflare did not give us that option or any other detail on the supposed issue.
If 90 or 95% of their traffic comes from a single domain (and presumably has for a while), that still doesn’t make OP sound guilty. If there was a legal issue Cloudflare legal should’ve stepped in, not their sales team.
That was the part that bugged me. This workflow is very busted from a user standpoint, though I'm sure it works very nicely to Cloudflare!
It smells like the "problem" was detected by automation, but instead of being able to reach anyone technical to work through it, you can only call sales teams.
In my opinion it's one racket vs another.
This shouldn't matter, in general Cloudflare responds to complaints about allowing illegal content with "we're a neutral utility, we forwarded your complaint to the site's webhost". To me, the article showed that Cloudflare was being extremely aggressive with selling the customer on an enterprise plan and repeatedly invented excuses to get them on the phone with their sales team. They then took the site offline and locked them out of their account when the customer started talking with other CDNs.
So the thing that stands out in the article, is that cloudflare's initial communications (and the final communications, when they moved to ban) implied issues with their behavior (trust and safety team, terms of service violations), but in between it sounds like the didn't talk about ToS at all, just sales team asking them to buy enterprise. Though it's possible OP is omitting some explanation given by as why enterprise plan would alleviate ToS issues.
>Hopefully Cloudflare provides a perspective.
Well HN is the unofficially official Cloudflare Support forum. I think we will hear from them soon. From past experience normally their response time for anything Cloudflare on HN is within 2-3 hours.
Except Cloudflare position here is not to ban them but they want to get paid for it. You are shaming the OP and his business but the reality is that Cloudflare has acted in a worse manner and that should be highlighted.
How does paying $10k a month solve that?
For $10k / mo paid 1 year in advance, your cloud provider does a legal review of the situation and figures out how to make your problem work on both the technical and legal level. It's not a "special plan", it's consulting.
Edit: "How do you know?" -- I don't know it's actually what happened, but when switching to enterprise, you don't go from 10% margin to 98% margin. The added costs actually represent added budget for the provider to deal with your "special case". ALL enterprise pricing tiers are disguised consulting contracts.
Or they had already decided to kick them out and tried to get some money out of them first.
Great theory!
The only questions that come to mind: how do you know? If that was the case why didn't they tell the customer?
It's 10k a month for them to set up a dedicated IP address pool so that they could BYOIP and buy their own IP addresses instead of getting the IP addresses in cloudflare's main IP address pool repeatedly banned or reputation harmed.
i.e. it's a $10k fee for maintaining the infrastructure for a quarantine around their services
Why can't they communicate than then? BYOIP also costs nothing to produce.
4 replies →
Nice place you got here. It'd be a shame if something happened to it.
Except the "place" isn't Mom and Pop's bodega, it's a casino dodging countries blocking its main domain.
Assuming that is what was happening, why would CF suddenly be okay with an illegal site if they pay more? Might as well call it the criminal enterprise plan then.
What are you trying to say here? You think extortion is ok if a country is trying to base itself where it is legal?
This is literal theory crafting lol. CloudFlare never said or implied that in their emails, yet you seem to know more than the CF reps themselves?
"Now this needs a bit of context on what they are talking about. We do have multiple domains that mostly act as mirrors to our main domain. We have these for a few reasons. One is that since we are a casino, we have different regulatory requirements we need to comply with in many countries. For example, many games are only available in some countries. Some countries we block completely. Then we have a few different domains that remove certain game groups or site features - for example our social features (chat, user tipping / interaction) or our sportsbook. Another is that we use them to target different global user groups and affiliates and track conversions long-term. This also means that if a country DNS-blocks our main domain, a secondary domain may still be available. This could arguably be seen as a violation of the Cloudflare TOS, as they wrote above."
Looks like they COMPLY with regulatory interest, to me.
When it comes to laws and taxes, "comply" and "evade" tend to be synonyms.
"In order to comply with tax regulations and donor laws, we had to structure our activities in order to make it possible for political donations to be classified as regular consulting income".
At least from discussions I've had over the years with my accountants, comply and evade are very different. Evasion is when you are doing something that's explicitly illegal. Optimization and compliance is when you comply with the law while trying as much as possible to reduce your tax. In some cases, there's a bit of a grey area where you use multiple structures that according to your accountant should comply with the law but in a way that has never actually been tested legally. That last part tends to be named "optimization" rather than "compliance"
1 reply →
If that's the take, that means Cloudflare is okay with 'breaking laws' so long as they can take a heavy cut of the ill gotten gains? </sarcasm>
Let's not try to find reasons to harm the messenger and stick to the facts -- a paying customer was suddenly extorted for hundreds of thousand of dollars out of nowhere.
Using localized versions of your services to comply with regional laws and enhance user experiences (i.e. make money) is SOP for practically every international $bigco. Online gambling is regulated and legal in ~50%-70% of the world; without actual evidence to the contrary, it’s completely reasonable to assume that this is a legitimate business. I’m really struggling to agree with the “two sides to every story” replies being left here about how there’s likely shady activity going on behind the scenes, when to me the post read as candid and transparent about the nature of the nature of the business, the admitted legitimacy of CF’s TOS violation claims against them, and the content of the communications with CF.
My 2c: It’s scummy that CF did this. It looks like they were disingenuous about the severity of the violations and used it as an excuse to get more $$$ from an already paying customer to make the manufactured problem go away.
It is a good article, good to have practical details of how this goes down... but really an international casino cant afford more than $250 a month?
Getting a demand to increase the payment by 40x is shocking no matter how much you make.
No sane company just goes “oh, that is fine. Must be that ‘ole inflation making times tough at $vendor, eh?”
Not in response to the way Cloudflare came at them, anyway.
Nah, you have different domains so you can track and maintain flows, also the regulations might even stipulate having domains in the locale, the headline is very much accurate after reading the article.
I mean, sure, they’re probably doing some sketchy regulatory dodging or whatever. Which part of this can Cloudflare solve by having them pay $120k/year to them?
The post mentions BYOIP. I assume Cloudflare wanted OP on BYOIP to mitigate risk, and Cloudflare wanted them to pay for the privilege.
The part where Cloudflare is happy to turn a blind eye to any “issues” if they get their $$$, apparently
Over-charging is a legal way to effectively deny service. When I'm offered jobs I don't want, I sometimes tell them my salary is 3x what I really need.
I feel like you would probably consult with a lawyer before saying you’d do assassinations for 3x your salary needs ;)
With a casino, the issue isn't just domains, it's also IPs. That's why they pushed BYOIP so heavily.
Obviously you work for CF. I did read the full post.
Yeah, there's some pretty key info being left out. I don't doubt that Cloudflare communication sucks especially when dealing with their sales team (aka bizdev which is what OP was originally contacted by), but the second screenshot is pretty damning.
My guess: Their account fell out of the non-enterprise TOS for some reason which is being obscured in the post (probably domain rotation related). Their T&S team proposed moving to enterprise for a custom resolution. OP's company refused, their account was purged because they had gotten several warnings about it.
I'm sure this sounds frustrating to the average HN dev who runs a legitimate startup with cloudflare on top and is now biting their nails worried to death about what will happen to them. But "online casino" immediately raised a million alarm bells in the post.
I did mention the multiple-domains issue in the post. It would not have been amazing for us to remove our secondary domains, but we would have been very happy to do it if it had resolved the issue. We asked them again and again but they would not give us any detail or options apart from their 120k/year package. Note that BYOIP (which I guess they could reasonably have required to isolate us even if we only use a single domain) is available for a fraction of the cost elswhere (e.g. fastly).
Since we already left Cloudflare the only reason I finished writing this article is to warn others. I think it's still relevant to many companies regardless of what you think of casinos, since very unprofessional sales tactics (unprofessional as in business threatening) seem common place with them. Do look at the linked other posts and comments here from other people affected that don't have anything to do with casinos.
I'm happy to answer questions as well.
For me, the worst part is blackmail and account ban.
If you had legal presence in EU then new Digital Services Act[1] might a help for you. I am not sure if you could sue them based on that law, but you could maybe lodge a complaint.
And the part where they offered to remove the secondary domains and couldn't get an answer?
Is the casino illegal in the jurisdiction they're based out of?
It doesn't seem so, so there is at least a valid reason for Cloudflare to keep them as a customer as they're not violating the laws where they have their business in.
This has been my experience with 80+% of these loud complaints about services, especially regarding "losing Google traffic". Dig into it just a little and you find out the complainer was doing something extremely shady that the service is often too polite/proper to call out in a public forum.