Comment by aurareturn
1 year ago
There are only two questions everyone have:
1. Would Netlify forgive the bill if this didn't go viral?
2. How do you plan to address this issue so that it never happens again?
Everyone here knew someone from Netlify would come and say OP wouldn't have to pay. That was a given. Now we want to know the important answers.
1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral
2. While I've always favored erring towards keeping people's sites up we are currently working on changing the default behavior to never let free sites incur overages
Any cloud platform should have a spend-stop amount built in.
i.e. if I know I average $10 a day, I should be able to put in a "If it hits $50, email me and take it offline".
Of course the opposite problem is then people setting that limit too low but since the user defines the limit that's on them not you.
This is one of the reasons I still in 2024 rent physical boxes and run the modern stuff on top of them directly, yes it costs me more per month but the price is hard capped.
This is something I really like about Nearly Free Speech.net. Their model is that you deposit funds up front, and they will deduct from those funds as you use services. It helps that they actually are nearly free so that a single $20 deposit can last for months or years in many cases.
It's bizarre to me that more services don't support billing this way, since there are tons of situations where I would much rather have a site or service go down than be hit with a surprise bill and have to depend on social media and magnanimous corporate PR.
22 replies →
We did this at DigitalOcean for similar reasons, wasn't a feature that was commonly used. Additionally, when you set that limit people then get upset because usually when they go over it for a good reason, like going viral, they aren't anticipating it, and just when their traffic is most valuable the site is down.
What Netlify is doing here is really the best approach for both parties. And typically speaking a $104k bill would be hard to get paid up regardless if the customer's typical transaction balance was $5/mo and their credit card limit wouldn't be that high.
Also, that's the benefits of credit cards - that you can still issue a charge back, and credit card companies very much favor the consumer rather than the merchant.
48 replies →
> I still in 2024 rent physical boxes and run the modern stuff on top of them directly, yes it costs me more per month but the price is hard capped.
I still prefer this too. Kinda funny how server resource limitations became a feature and not a bug when it was one of the problems the cloud sought to overcome
2 replies →
Yes, any price that's not hard capped is unacceptable. One reason why I quit Amazon cloud after the trial year. No because they were too expensive, but there was no way to guarantee they wouldn't charge me more than planned...
If a cloud platform offers such a limit, but the user fails to set it up, then uses $100,000 of bandwidth, is the platform then justified in NOT forgiving the bill?
If forgiving bills for this kind of a thing is a standard practice, how come this was the customer support's first reaction:
>We normally discount these kinds of attacks to about 20% of the cost, which would make your new bill $20,900. I've currently reduced it to about 5%, which is $5,225.
20% and 5% are quite a bit higher than forgiven.
Given this has been asked here multiple times without response from /u/bobfunk, it’s hard to conclude anything except that he is lying.
4 replies →
This needs to be much higher up.
> 1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral
This isn't what you said in your first post, you said:
> It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.
So forgiving "lots and lots" doesn't move the needle. Do you or do you not forgive _all_ such cases where your DDOS protection doesn't take down the site? What was your employee referring to when saying that the usual discount is 20%? Are you saying that you _never_ discount 20% and instead always discount 100% i.e. "forgive"?
I don't think this will get answered, unfortunately.
1. Forgiven many, is Netlify forgiving all obvious anomalies? Is the question, which if so but you said many so it is a no, it would make you reconsider the next point 2. Favoring keeping people site up ? Would you go as far as keeping them up if they stopped paying for the meter? If not you simply should not let that meter go overboard.
Hey I'm a taxi driver. Hailer fell asleep on the back, so I kept driving all night, once he woke up I dropped him to his place and asked for my monthly wage. I "forgive" many, but just a few are juicy income so I adopted the policy to never wake any customer up. If people ask I say it would be impolite, principles prime.
Regarding #2: I would rather have my hobbyist website go down rather than facing the daunting task to raise a query on HN and hope the bill goes away.
> 1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral
Sequence of events doesn't support this answer:
1. User gets charged 100k
2. User complains to support
3. User receives discount to 20k, then 5k. Support states policy is normally 20k
4. User discloses to the world. Goes viral.
5. Invoice is forgiven
While you might forgive "lots and lots", fact is that you still presented the invoice to a free tier customer, and when they complained you gave them a discount, but still charge. Only when it went viral did you forgive it.
Quite... It does seem that either the story we're getting isn't completely accurate or the support people who handled this need a little reminder of what's supposed to happen.
I'm a paranoid person by nature so "It's free... just... give us your card details" is always suspicious.
1 reply →
They had forgiven lots and lots of bills, but forced to pay lots and lots more people.
Give that there are free stressers/booters , and reasonable prices to rent a DDoS cloud.... https://stresser.su/#pricing
1. What are you doing to prevent DDoS's from hitting your network?
2. Why do customers have to allow an unlimited credit burden to use services?
3. Why arent there cost controls to "if $$ exceeds X, shut acct down"? Azure can do this.
Long story short, why are you by default (except for social media escalation) passing fraud costs to customers?
But you realize that a small business or startup can't rely on "generosity" to avoid going bankrupt?
It seems that significant bills appearing without warning or cut-offs is clearly intentional. I am embarrassed that I recommended Netlify before.
Do the changes you are working on that will cause "the default behavior to never let free sites incur overages" involve providing users with spending limit controls?
Solving this only for the free site use case doesn't address the core problem that people are bringing up about a lack of spending limit controls.
Do we have anything more binding than your word to rely on?
From what I see you could change this policy tomorrow unilaterally and we would have no recourse.
> this policy
I wouldn't think it's a binding policy at all, because the billing procedure (automatic full bill, manually discounted bill, etc.) would follow it if it were. More of a procedure.
Why did this happen in this case if you said it doesn't? Netlify fought to bill OP repeatedly until it went viral.
> 1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral
No offence, but this sounds like "trust me bro" billing and it is not good enough. Someone could literally get a heart attack from getting $100,000 bill - this amount of debt can literally ruin someone financially.
> 2. While I've always favored erring towards keeping people's sites up we are currently working on changing the default behavior to never let free sites incur overages
I hope you understand that chance someone who used to pay you $20 / month unlikely want to ever get $10,000 bill. Yeah people might dislike that their website went down due high traffic, but it's not gonna bring this much negative PR as incidents like this. There should be some sanity check at least.
> Someone could literally get a heart attack from getting $100,000 bill - this amount of debt can literally ruin someone financially.
Not a heart attack but there has literally been at least one suicide over an unexpected life-ruining bill like that:
https://www.forbes.com/sites/sergeiklebnikov/2020/06/17/20-y...
2. is obviously what should have always been the case, but it's good news to hear you've now gotten there. Every single hobbyist website would always choose downtime over a hundred thousand dollar charge.
With a properly configured nginx, you can easily serve 10's of thousands of requests a second on vserver type hardware. Netlify just offers these build pipeline kind of static site with cms UI.
But this is a good reminder why my gut feeling always made me avoid these overengineered solutions.
2 replies →
You should probably consider a daily limit (up to some max n days) rather than a hard one time limit. If your engineers can set a 1 and done they can set an n and done and it would be a much better solution and more customer friendly. The guy using 5 gigs today as a poor college student will likely have a position in a small to mid-size company in a few years. I assume non-free (but low tier) customers would much prefer a reasonable limit set as well. Maybe a max of 2x (or so ) bandwidth so no huge surprises. Remember they're your customers and not your paying adversaries
I’m sorry. You are working on changing things so FREE sites don’t get charged???
That’s the elephant in the room here. I understand an enterprise plan where you state billing is $xx per GB, but billing someone with a free site??
Give me a break.
This seems like a really good idea to me. Or at least cap overages at a specified amount, like 2x the free level (a $55 surprise bill is a totally different universe from a $100,000 surprise bill, obviously).
Honestly, this terrifies me---I run a bunch of different sites off netlify, and I would have never imagined that a site could jump from 0 to six figures of bills a month without something hitting a tripwire somewhere and cutting it off or at least communicating with the account owner. At least users should have the capacity to self-impose bandwidth caps to prevent this sort of thing.
"wOrKiNg oN"
Thank God for social media that the user was able to get attention about this on Reddit which he was then advised there to post this on HN. It must have been stressful to see a six-figure bill and then get told that that, no worries, you’d ‘only’ be charged $5k instead for a static site. It’s just ridiculous to me to be sent a 6-figure bill in the first place.
I hope this is not one of the cases that get simply forgotten and in a week or two their beginner unfriendly platform gets recommended again without a second thought.
With models like this and AWS people will get afraid of success
I think fly/netlify/vercel/render etc. get a decent enough flak on here for costs and/or reliability.
The average HNer seems to be recommending colocating your physical server :-)
1 reply →
Well, it's still debatable for the history books if social media is a net good.
Before the internet, these issues would be handled by local news journalism, and still sometimes do!
I mean, social media is pretty much an inevitability once mobile phones/internet became mainstream. Just like the invention of the gun and gunpowder, I think we are still debating if it was good for society right to this day.
From the 5% reduction it seems (1) was less likely.
To bobfunk, the response needs more empathy and explanation around the obvious frustration around why there is no slider for cost limitation.
As it is, it feels like the minimum viable corpspeak apology and damage control.
You don't see VPS providers like Vultr forgiving bills like this, nor do they make the news. Granted they are not the same scope as Netlify, but still.
OP said they agreed to reduce the payment, which means they acknowledged it was an attack but still wanted payment
if only i had $1 for every time for every time someone asked this exact question on HN. yes, we all get it: easy question is askable and not answerable. you want a gold star?