Comment by skrebbel
8 months ago
We had a similar experience. Stuff suddenly started breaking for 10% of our traffic, support dragged their feet for weeks wrt any sort of insight as to what was going on, and then the answer was “you’re over an undocumented limit, try the enterprise plan”.
Fwiw this was some years ago and we moved most of our stuff away from them in response. I didn't get the feeling that this was malicious from their side, more like growing pains / mediocre support people / etc. But the end result was the same as you describe, except we chose not to pay up.
EDIT: more context: I shared this story on HN once before, jgrahamc responded with “please email me”, we did but it didn't move the needle. This further convinced me that CF just has a lot of stuff going on and something weird about our traffic made them error out. My suspicion is that the enterprise plan was supposed to make it internally defensible to pour more engineering resources into our case, but they were never explicit about that which made us worry enough to not do it.
I think that a large reputable business like CF should be clearer about stuff like this. That said, as someone running an API business, I also hold some sympathy for “customer does something weird an unexpected, it’s hitting a limit we didn't even know we had, srsly now what?”. The answer to that should be “work together with the customer to get to the bottom of things, customer might need to make changes too”. They didn't do that, which disappointed us, but I can relate to the situation nonetheless.
We’re still a CF customer, just not for this part of our offering.
> undocumented limit
this makes it sound like the limit is automatic or applies non-discriminately to customers, but my first instinct is that this was manually set by someone, maybe the sales reps again?
Yeah so I think it might’ve been a real system limit of sorts. Something timing out somewhere, some pipe getting clogged in a way that their edge nodes couldn’t scale their way out of the way they usually do. Eg because the scaling/monitoring code didn't detect that particular pipe getting clogged etc. We had weird long-running http requests at the time.
Note, this is pure conjecture, I’m just well aware from my own engineering experience that stuff can break under varied load in all kinds of unexpected ways. A large part of the work of an infrastructure business is going “woa shit I hadnt expected that we could fail in that way too” and then building infrastructure to be able to handle that case. You simply can’t predict everything your customers are going to throw at you. I think this was what happened + not sufficiently knowledgeable/experienced support. But I admit that I’m really just guessing.
The alternative would be that CF purposefully dropped 10% of our traffic to convince us to upgrade to enterprise, and despite our bad experience, I don’t believe they’re that kind of business. And if they were they handled it very badly because it took them 3 weeks of feet-dragging to even bring up the upsell.
> I don’t believe they’re that kind of business
I didn’t either, but then I read this post :/
1 reply →
I've hit one of these undocumented limits before where stuff would randomly start to fail. Once I was able to get a sales rep to talk to me about it, the problem suddenly went away. I didn't even need to buy an enterprise plan, but would have had they asked me to do so.