Comment by pclmulqdq
8 months ago
As far as I can tell, the issue with this is:
OP runs a casino/gambling site. Gambling is a regulatory mess (I have spent far too long dealing with this as an RNG supplier), and so it's very hard to comply with every jurisdiction, and each one needs you to prove compliance to operate in that jurisdiction.* Gaming companies spend a lot on compliance and tracking, but since the internet is the internet, it's pretty hard to enforce perfectly, so some countries and ISPs take this into their own hands.
Due to that, IPs hosting gambling and gaming sites often get regionally blocked by internet providers or otherwise flagged as hosting illegal content. Those regional blocks consequently affect the reputation score of the IP, and if you are a traffic aggregator like Cloudflare, can cause other customers to have issues. One of the most aggressive and annoying regulatory environments for gambling companies is the US, so it's very possible Cloudflare has had some trouble due to gambling use of their IPs in states in the USA.
Cloudflare wanted them to use the BYOIP features of the enterprise plan, and did not want them on Cloudflare's IPs. The solution was to aggressively sell the Enterprise plan, and in a stunning failure of corporate communication, not tell the customer what the problem was at all. The message from Cloudflare should have been "Enterprise plan + BYOIP or ban, and maybe we'll work with you on price" but it was instead "you would really like the Enterprise plan."
*As an aside - we're lucky in that respect being a tech supplier with relatively uniform rules, but our customers (the gaming companies) get the short end of the stick here.
BYOIP is reasonable, though I doubt anyone actually does legislation blocks by IP. Since like half of companies on the internet use Cloudflare or other multi-tenant infrastructure everyone is aware that you can't block an IP address and hit one target. The only thing I've seen is DNS blocks (both DNS protocol directly and based on TLS SNI).
FYI, we also fully block users from the US (due to regulations).
My problem here is mainly the unprofessional communication and huge mess of mixing "compliance" with sales, without giving any clear information or options. And then the removal of our account without warning while we were still talking to them.
You would be surprised how big of a hammer ISPs will use when they are told to hit something. They live in a very different world than many modern web software companies - they are the plumbers for lots of things you take for granted, and look at the world the way a plumber does. Thanks to TLS, the plumbers can't see the HTTP headers to figure out what's actually flowing, so they sort of end up whacking all of it.
Generally, low-reputation IP addresses are associated with scams, spammers, and other similar things. Gaming somehow gets lumped into this bucket in some jurisdictions, but that hurts you worldwide (similar with other "sin businesses" like porn). These blacklists get published (I think there's some parts of BGP that make this happen, but I'm not quite sure what the mechanism is), and being on any one of them hurts your traffic everywhere because it becomes suspect.
I agree with you that this mix of compliance, engineering, and sales is gross. If this was the issue, they should have just told the OP.
It will be interesting to see. Just for completeness, Fastly is not requiring us to BYOIP or anything unless it causes them actual problems, which so far it hasn't. I'm sure they also have other similar businesses to ours so they should have some experience.
I guess I'll see in a while if this was also just a sales tactic from Cloudflare or not.
1 reply →
While they absolutely shouldn't ban IP for reasons you said, some do that anyway.
The most (in)famous case is China's GFW which banes IPs all the time. Yes, other websites often get accidentally blocked, but they don't care. Moreover, you can't even communicate with them because there are no official legal regulations. This is something what any CDN or cloud providers have to deal all the time.
There are numerous official legal regulations and decisionmaking bodies pertaining to the GFW. What do you mean?
1 reply →
>The solution was to aggressively sell the Enterprise plan
A demand for 120k upfront or else bad stuff happens is by no reasonable definition "selling", aggressive or otherwise
Sounds like something Sammy The Bull would be sent to do, not the sales team.
I also wonder if the company in the article didn’t know that (either by reading between the lines as you did or via other correspondence they didn’t mention) and weighed that in their decision to go with Fastly.
BYOIP isn’t just expensive—if your content is bad for IP reputation the time-to-flagging of your IPs is going to be way shorter on BYOIP than on shared IPs due to there being less dilution. And that’s without getting into the challenges around rotating/renting IPs on a continual basis.
I do agree that CF did not communicate that well or professionally—if the sales emails are the only comms that happened here.
I think it is possible that the company posting this didn't realize that this might be the issue, but you are right that they may know. It may have been a small company, even doing that much bandwidth. Online gambling sites tend to push an entire video game when you are playing on their site.
Many gambling companies are fine just doing BYOIP or running dedicated hosting infrastructure that is on providers who are explicitly running hosting for that industry (although they are moving to cloud). There is a good reason this separate infrastructure exists. In general, I would not assume they are rotating IPs: this is not a scam, it's a business, and they are largely fine with being blocked in places where they can't legally operate.
Maybe the first couple of weeks there was a misunderstanding, but by May 7 it was clear what the situation was. They was told they need BYOIP through and enterprise plan; they just didn't want to pay for it.
>I have spent far too long dealing with this as an RNG supplier
Is there much of a market for that? I thought random.org had it all sewn up.
No, random.org isn't in that market at all, aside from doing some drawings local to them and unofficial games.
I think we are only one of ~3 TRNG suppliers who have been audited. Many games don't use a TRNG, though.
Since it uses atmospheric noise, you can also influence the numbers from random.org by transmitting radio waves in the area nearby - the operator of random.org has mentioned that there's so much RF activity that he is concerned about whether the bits are still random. A final issue is that they are also so low-volume that they probably can't get enough test data for the required audits (which can be a lot of data).
To underscore the volume question: Random.org used to have a running count of bits generated. The counter wasn't monotonic (before it broke in ~2015-2019), but the peak value I saw when I checked archive.org was about 250 GiB total since 1998 (that was in 2015). That is one quarter of the size of our "light" qualification test ("heavy" is 16 TiB). The RNG auditors also take O(100) megabytes for each audit, which would be a significant fraction of random.org's output.
honestly this is fascinating to me, I was curious too and upon searching "RNG Supplier" I couldn't find anything, 3 supplies in the whole world is a crazy supply-side industry!
I was just curious to see what a landing page of a RNG supplier looked like, how do you even do sales for such a thing? With 3 players I guess it's just something you know in the industry and those partnerships are likely long-lived, right?
niches like this fascinate me for some reason!
> Is there much of a market for that?
I don't think it's a huge market, but state-run lotteries around the world need good random number generators for games without physical balls (like Keno for example).
I've talked with people that have created RNGs (rather than buying off-the-shelf solutions) and it sounds like soul-crushing work - mostly due to dealing with the government regulators that need to give final approval before the RNG can actually be put into production.
Gaming machines are highly regulated, almost like medical devices.
There are seals on the hardware, any modification must be approved, you must certify that the payout is the expected one, ...
I'd say worse than medical. As long as it works and doesn't fail, no one will give a fuck about medical hardware or the condition it's in. Just look at your average GP's ultrasound, it's probably older than the GP themselves is.
Gambling however, you constantly have government auditors and the tax office crawling up your literal arse to make sure you don't cheat the gamblers, or worse, the government out of their money. And in some cases, add the mob or other criminals on top who also want their cut.
4 replies →
It's worse than medical devices, at least for the final machines and software.
For components in the path of money flow (payment processors, RNGs, hosting, etc.), it's similar.
Ok, I have to ask: are you a Random Number Generator (RNG) supplier, or a Renewable Natural Gas (RNG) supplier, or some other kind of supplier??
Considering they mentioned working with gambling/casinos, I would assume random number generator. Which may seem somewhat trivial to build a business around, except if you’re in a highly regulated industry like gambling that regulates the implementation of randomness (and probably requires auditing and other complicated things like that). I would love to read a blog post on all the complexities at here.
It almost sounds like you are excusing them. Asking people to switch to the enterprise plan and bring your own IP is reasonable, but not on a timeline of 24h, and trashing their account when they tell you they are talking to a competitor makes me feel like I should flee Cloudflare services with all haste.
> and trashing their account when they tell you they are talking to a competitor
That is just a story they have made up. They don't know why Cloudflare shut their account down. I reckon the Fastly "reason" is likey a red herring.
It does not really matter, the 24h deadline is incredibly unprofessional. And deleting their account even more so.
I think the lesson here is to be as provider agnostic as possible and have a backup plan in case your current provider decides they don’t like you anymore or they just delete all your data just because they can.
This is just how people communicate now in the business world. It's up to you to read between the lines.
One thing I've learned to be wary of on the job is "do you need help?" That phrase is often code for "You are not performing up to our expectations. This is your first and only warning. Get in shape or get out."
Poor communication is worse than no communication.
[dead]