Comment by bobfunk
1 year ago
1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral
2. While I've always favored erring towards keeping people's sites up we are currently working on changing the default behavior to never let free sites incur overages
Any cloud platform should have a spend-stop amount built in.
i.e. if I know I average $10 a day, I should be able to put in a "If it hits $50, email me and take it offline".
Of course the opposite problem is then people setting that limit too low but since the user defines the limit that's on them not you.
This is one of the reasons I still in 2024 rent physical boxes and run the modern stuff on top of them directly, yes it costs me more per month but the price is hard capped.
This is something I really like about Nearly Free Speech.net. Their model is that you deposit funds up front, and they will deduct from those funds as you use services. It helps that they actually are nearly free so that a single $20 deposit can last for months or years in many cases.
It's bizarre to me that more services don't support billing this way, since there are tons of situations where I would much rather have a site or service go down than be hit with a surprise bill and have to depend on social media and magnanimous corporate PR.
Yes it’s nice like that. Specially for side projects on AWS that could go wrong on your personal credit card. Also I heard they forgave bills sometimes.
6 replies →
I absolutely love NFS and pay for the support option even though I don’t need it, simply to support them. Everything about that service is simply ace.
A couple years ago I switched from a standard, contract-based US cell phone plan to a prepaid service it has felt so much more natural. Have a messed up my autopay and my phone just stop working in the middle of the day? Yep. But You just find some WIFI, pay, and its back on in minutes. I know exactly what my service costs: $35 dollars. No fees, taxes.
12 replies →
I think for this purpose you could use something like privacy.com, generate a virtual credit card, and set a monthly limit on it. This way you don't have to deposit any money in advance.
We did this at DigitalOcean for similar reasons, wasn't a feature that was commonly used. Additionally, when you set that limit people then get upset because usually when they go over it for a good reason, like going viral, they aren't anticipating it, and just when their traffic is most valuable the site is down.
What Netlify is doing here is really the best approach for both parties. And typically speaking a $104k bill would be hard to get paid up regardless if the customer's typical transaction balance was $5/mo and their credit card limit wouldn't be that high.
Also, that's the benefits of credit cards - that you can still issue a charge back, and credit card companies very much favor the consumer rather than the merchant.
> Also, that's the benefits of credit cards - that you can still issue a charge back, and credit card companies very much favor the consumer rather than the merchant.
So your suggestion is to issue a chargeback.. to get money back that should under the terms of whatever service you signed up for be owed?.
That seems like bordering on fraud tbh.
> Additionally, when you set that limit people then get upset because usually when they go over it for a good reason, like going viral, they aren't anticipating it, and just when their traffic is most valuable the site is down.
Legit concern and something I mentioned, I'm gonna guess there are broad two camps on that one - mine which is "I want a safety ripcord" and "whee, nice problem to have".
However since this entire conversation is around a guy who got a massive invoice because of a bill he wasn't expecting and couldn't have set such a limit I'm still gonna go with a "I want a way to constrain the financial downside - hell turn it off by default but give me the option".
Since broadly a lot of cloud stuff doesn't, I'll constrain it a different way.
2 replies →
> Also, that's the benefits of credit cards - that you can still issue a charge back, and credit card companies very much favor the consumer rather than the merchant.
That has not been my experience. I've had to do a few chargebacks for services not rendered, and I've never won. I will submit my evidence, then the vendor will submit 100 pages of random emails, and then I will have my claim denied. Then I will appeal, will point out that they sent 100 random pages of email, and then they will reply with the same 100 pages of emails and I'll get denied again.
It seems that the vendors have found the hack for chargebacks -- just inundate the credit card company with so much data that they assume the vendor must be right.
It makes sense -- the vendors pay the credit card companies a lot more than I do. They'd rather keep them happy than me.
2 replies →
But that's on the user. The user shouldn't get upset in that scenario and has no right to. You're giving the control back to the user.
4 replies →
A debt is still a debt even charged back.
There would be many attorneys interested in collecting a $105k debt.
11 replies →
Here's an idea: let users set a spending limit.
If they're about to go over, shoot them a quick heads-up and give them 24 hours to sort things out or level up their package.
After that, if they haven't made any changes, temporarily pause their site access
11 replies →
That sort of adversarial weaponization of credit cards is really a US thing. In most of the world, payment is not relevant to the question of liability: you create a debt, whether or not you can debit a card. So, looks like DigitalOcean is another name to avoid.
Also: what's common? At the scale these companies so desire, small percentages are still thousands of users. Supporting them with what's ultimately a fairly trivial option shouldn't be seen as such a hassle.
3 replies →
Just want to chime in:
1) Thank you for allowing us to set the limit.
2) I understand your opinion that you prefer chargebacks but I disagree with it.
The very reason I stay with Hetzner is that I know in advance what my bills will be for the whole year. Heck, I even charge my account in advance so that I don't worry about any charges!
>that you can still issue a charge back
Most users don't want to be banned from their hosting provider.
> usually when they go over it for a good reason, like going viral, they aren't anticipating it, and just when their traffic is most valuable the site is down
Of course. And that’s why any limit against a dynamic variable should also have alerts linked to it
Send an alert to the user when traffic starts spiking, especially if a simple projection shows it’s going to go over their limit
Then the user is aware, hopefully with enough time to lift the limit if needed
2 replies →
> What Netlify is doing here is really the best approach for both parties.
Sort of, but the approach to the approach isn't great. If you're going to void charges from DDOS traffic anyway, you might as well make that explicit policy, rather than doing it after the fact in a way that seems discretionary.
The Reddit thread is full of people who are saying that they intend to pull their static content off of Netlify and move it to Cloudflare Pages, which has no overages on the free tier in the first place. I can tell you that I personally chose Cloudflare Pages to set up a new static site a couple of weeks ago, and Netlify wasn't even in the running.
If Netlify's free tier is important to their customer acquisition strategy, then they really need to retool how they're offering it, or Cloudflare is going to eat their lunch.
1 reply →
It seems like that could be an option like:
Hard limit: [$1000]/mth or [$500]/24hr period
Notify me if traffic exceeds thresholds: [$800+]/mth or [$400]/24 hr period
Notify me if the traffic forecast for the month looks like it will exceed my hard limit before the end of the month.
They still need to make tweaks to avoid sending $100k bills to people who signed up to a free service.
Like let them go over one month and then make them sign up for a paid plan for the next.
> I still in 2024 rent physical boxes and run the modern stuff on top of them directly, yes it costs me more per month but the price is hard capped.
I still prefer this too. Kinda funny how server resource limitations became a feature and not a bug when it was one of the problems the cloud sought to overcome
Physical Boxes at Hetzner and the likes are usually way cheaper than most cloud offerings
1 reply →
Yes, any price that's not hard capped is unacceptable. One reason why I quit Amazon cloud after the trial year. No because they were too expensive, but there was no way to guarantee they wouldn't charge me more than planned...
If a cloud platform offers such a limit, but the user fails to set it up, then uses $100,000 of bandwidth, is the platform then justified in NOT forgiving the bill?
If forgiving bills for this kind of a thing is a standard practice, how come this was the customer support's first reaction:
>We normally discount these kinds of attacks to about 20% of the cost, which would make your new bill $20,900. I've currently reduced it to about 5%, which is $5,225.
20% and 5% are quite a bit higher than forgiven.
Given this has been asked here multiple times without response from /u/bobfunk, it’s hard to conclude anything except that he is lying.
I will put there the other obvious offender: Vercel . Not sure about bandwidth, but dark patterns, keeping serious RBAC procedures we'll hidden until asking a fortune even for startups, to provide not things like SSO, just reasonable RBAC.
With all that money they then can finance the free tier until they get too far and become platform locked-in.
Surge.sh Im not sure. But shows all the sign of some greedy acquisition, regular long outages , as if I have been sitting as a free tier for too long, quick nudge to pay. For barely accessed sites even behind CDNs, steep. I even worry they one day just wipe all my buckets (they did for a few already) and support would recommend me to be a "normal" paying user .
Nothing is free. And nothing too good to be true is true .
Or that usage and billing is a difficult space and this particular thread has been dogpiled by a bunch of folks that have never actually worked in it.
1 reply →
Lying but a good talker for sure.
I wouldn't want to be CEO these days. A lot are trained and paid to do damage control.
This needs to be much higher up.
> 1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral
This isn't what you said in your first post, you said:
> It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.
So forgiving "lots and lots" doesn't move the needle. Do you or do you not forgive _all_ such cases where your DDOS protection doesn't take down the site? What was your employee referring to when saying that the usual discount is 20%? Are you saying that you _never_ discount 20% and instead always discount 100% i.e. "forgive"?
I don't think this will get answered, unfortunately.
1. Forgiven many, is Netlify forgiving all obvious anomalies? Is the question, which if so but you said many so it is a no, it would make you reconsider the next point 2. Favoring keeping people site up ? Would you go as far as keeping them up if they stopped paying for the meter? If not you simply should not let that meter go overboard.
Hey I'm a taxi driver. Hailer fell asleep on the back, so I kept driving all night, once he woke up I dropped him to his place and asked for my monthly wage. I "forgive" many, but just a few are juicy income so I adopted the policy to never wake any customer up. If people ask I say it would be impolite, principles prime.
Regarding #2: I would rather have my hobbyist website go down rather than facing the daunting task to raise a query on HN and hope the bill goes away.
> 1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral
Sequence of events doesn't support this answer:
1. User gets charged 100k
2. User complains to support
3. User receives discount to 20k, then 5k. Support states policy is normally 20k
4. User discloses to the world. Goes viral.
5. Invoice is forgiven
While you might forgive "lots and lots", fact is that you still presented the invoice to a free tier customer, and when they complained you gave them a discount, but still charge. Only when it went viral did you forgive it.
Quite... It does seem that either the story we're getting isn't completely accurate or the support people who handled this need a little reminder of what's supposed to happen.
I'm a paranoid person by nature so "It's free... just... give us your card details" is always suspicious.
The issue is that you don't have to give the credit card details in this case. That charge wouldn't probably have gone through anyway.
They had forgiven lots and lots of bills, but forced to pay lots and lots more people.
Give that there are free stressers/booters , and reasonable prices to rent a DDoS cloud.... https://stresser.su/#pricing
1. What are you doing to prevent DDoS's from hitting your network?
2. Why do customers have to allow an unlimited credit burden to use services?
3. Why arent there cost controls to "if $$ exceeds X, shut acct down"? Azure can do this.
Long story short, why are you by default (except for social media escalation) passing fraud costs to customers?
But you realize that a small business or startup can't rely on "generosity" to avoid going bankrupt?
It seems that significant bills appearing without warning or cut-offs is clearly intentional. I am embarrassed that I recommended Netlify before.
Do the changes you are working on that will cause "the default behavior to never let free sites incur overages" involve providing users with spending limit controls?
Solving this only for the free site use case doesn't address the core problem that people are bringing up about a lack of spending limit controls.
Do we have anything more binding than your word to rely on?
From what I see you could change this policy tomorrow unilaterally and we would have no recourse.
> this policy
I wouldn't think it's a binding policy at all, because the billing procedure (automatic full bill, manually discounted bill, etc.) would follow it if it were. More of a procedure.
Why did this happen in this case if you said it doesn't? Netlify fought to bill OP repeatedly until it went viral.
> 1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral
No offence, but this sounds like "trust me bro" billing and it is not good enough. Someone could literally get a heart attack from getting $100,000 bill - this amount of debt can literally ruin someone financially.
> 2. While I've always favored erring towards keeping people's sites up we are currently working on changing the default behavior to never let free sites incur overages
I hope you understand that chance someone who used to pay you $20 / month unlikely want to ever get $10,000 bill. Yeah people might dislike that their website went down due high traffic, but it's not gonna bring this much negative PR as incidents like this. There should be some sanity check at least.
> Someone could literally get a heart attack from getting $100,000 bill - this amount of debt can literally ruin someone financially.
Not a heart attack but there has literally been at least one suicide over an unexpected life-ruining bill like that:
https://www.forbes.com/sites/sergeiklebnikov/2020/06/17/20-y...
2. is obviously what should have always been the case, but it's good news to hear you've now gotten there. Every single hobbyist website would always choose downtime over a hundred thousand dollar charge.
With a properly configured nginx, you can easily serve 10's of thousands of requests a second on vserver type hardware. Netlify just offers these build pipeline kind of static site with cms UI.
But this is a good reminder why my gut feeling always made me avoid these overengineered solutions.
It wasn't the engineering that did it, it was the egress charges. But I think the IaaS tend to charge more for this than the clouds, who in turn charge more than the "servers for rent" people.
They aren't engineered, they subsidize (more) enginnering effort , and (are meant to) cost less as a result.
They do. But of course maximizing profit is the sole true prerogative of capitalist enterprises. And the market is not totally competitive. So yes your intuition was correct, to be cautious against over enginnered pricing to get y'a.
I mean those companies cater to hobbyist. Then ...
Render seems more fair-play. Until a change of mgt occurs of course.
You should probably consider a daily limit (up to some max n days) rather than a hard one time limit. If your engineers can set a 1 and done they can set an n and done and it would be a much better solution and more customer friendly. The guy using 5 gigs today as a poor college student will likely have a position in a small to mid-size company in a few years. I assume non-free (but low tier) customers would much prefer a reasonable limit set as well. Maybe a max of 2x (or so ) bandwidth so no huge surprises. Remember they're your customers and not your paying adversaries
I’m sorry. You are working on changing things so FREE sites don’t get charged???
That’s the elephant in the room here. I understand an enterprise plan where you state billing is $xx per GB, but billing someone with a free site??
Give me a break.
This seems like a really good idea to me. Or at least cap overages at a specified amount, like 2x the free level (a $55 surprise bill is a totally different universe from a $100,000 surprise bill, obviously).
Honestly, this terrifies me---I run a bunch of different sites off netlify, and I would have never imagined that a site could jump from 0 to six figures of bills a month without something hitting a tripwire somewhere and cutting it off or at least communicating with the account owner. At least users should have the capacity to self-impose bandwidth caps to prevent this sort of thing.
"wOrKiNg oN"