← Back to context

Comment by ji_zai

1 year ago

This is my worst nightmare as a bootstrapped founder. And that there's no way to put a limit on spend is ridiculous. Someone that doesn't want me to do well can simply ddos me into bankruptcy out of nowhere.

Just went through Vercel's docs:

---

"Vercel helps to mitigate against L3 and L4 DDoS attacks at the platform level. Usage will be incurred for requests that are successfully served prior to us automatically mitigating the event. Mitigation usually takes place within one minute.

Usage will be incurred for requests that are not recognized as a DDoS event, such as bot and crawler traffic.

You should monitor your usage and utilize Edge Middleware to protect against undesired traffic based on its IP, User-Agent header value, or other identifiers."

---

That doesn't help me sleep well.

I feel that by now, these hosting providers should simply adopt best ddos protection practices and take responsibility for failure to protect.

"You should monitor your usage and utilize Edge Middleware to protect against undesired traffic based on its IP" - there should be some really good defaults for this right?

Clearly it's possible - Cloudflare's ddos protection is worded more strongly.

I'm willing to pay more for the service for peace of mind. Like, even $10/mo more to insure against getting smacked out of nowhere.

> Cloudflare's ddos protection

Yeah, we got hammered once with over 10TB/mo and noped out of Netlify as fast as we could: https://twitter.com/rethinkdns/status/1370342245841342466 Had to pay the bill in full.

Cloudflare's free tier is ridiculous: We do over 30TB+ of genuine traffic for $0. Makes it hard to move to any other platform. As a small tech shop, this is my Hotel California I'm happy to never leave.

  • CloudFlare pricing is indeed positively ridiculous.

    At OpenTofu[0] we’re using CloudFlare R2 to host the providers and modules registry[1]. Bandwidth is free, you only pay for requests.

    This already would be great, but there’s more - you only pay for requests that actually hit R2. So with an almost 100% cache hit ratio, we barely register any billable requests.

    Recently someone decided to load test us and generated ~1TB of traffic over 1-3 days. All but a few of these requests were cached, so the whole situation probably cost us less than a cent.

    [0]: https://opentofu.org

    [1]: https://github.com/opentofu/registry

    • Is this in line with the TOS? I thought there were restrictions on serving non-website content in the free tier, or does that not apply to the CDN if you're using R2 as an origin?

      2 replies →

  • > Cloudflare's free tier is ridiculous: We do over 30TB+ of genuine traffic for $0. Makes it hard to move to any other platform. As a small tech shop, this is my Hotel California I'm happy to never leave.

    Yeah that's how Cloudflare can reach total control over the Internet. With thunderous applause by people that should know better.

    I know that my position is outright blasphemous in this day and age, where even self-hosting a static site has become black magic and we need a third party to do it for us.

    • I don't understand this take. First of all, moving off of Cloudflare is trivial if you really have an alternative. Second of all, self hosting a static website is easy, but that's not we're talking about here. We're talking about DDoS mitigation, which is not gonna be solved over a weekend hack with a load balancer. At least, not at the scale that matters.

      What would the Cloudflare going evil phase even look like? Is it anything like Netlify charging me 100k because they don't provide ANY DDoS protection? I don't see any FOSS tools preventing this problem.

      2 replies →

    • > Yeah that's how Cloudflare can reach total control over the Internet. With thunderous applause by people that should know better.

      This is an emotionally-manipulative, anti-intellectual comment that certainly does not belong on HN. There's no intellectual curiosity or value in this comment - just scoffing, predictions of doom, manipulative statements like "I know that my position is outright blasphemous in this day and age", and other drivel that belongs on Reddit, not here.

  • That’s a free tier that doesn’t sound sustainable then, so that raises alarm bells to me.

    • Peering.

      Here's how it works:

      1) I have a big network and I exchange traffic with another big network. Think of "eyeball" networks like last-mile ISPs (Comcast, mobile providers, etc) where a substantial portion of end-user traffic is going to handfuls of well known networks - Cloudflare, AWS, Netflix, etc.

      2) Comcast and Cloudflare say "Hey, I send you X TB/PB/etc and you send me X TB/PB/etc. We both currently pay another provider to route that traffic between us. Let's not do that."

      3) In locations where it makes sense they basically throw a cable across datacenters, POPs, internet exchanges, etc. The cost for this is typically extremely low - it's basically a port on a switch/router on each side and MAYBE a "cross connect fee" from the facility. This is usually billed in the tens of dollars/mo if at all. It takes very little time/effort to configure this but of course the details are more complex - multiple ports, multiple facilities, etc.

      4) Both sides start routing traffic between their networks over their new shiny direct cables and extremely high speed ports. Faster throughput, lower latency, improved reliability, frees up bandwidth to the transit provider they were using previously, and most importantly the cost of bandwidth between the two networks goes to zero.

      This is all well known and publicly available because it's visible in the global routing table(s). Cloudflare, for example[0].

      All of the large providers do this and AWS, etc charging in bandwidth per GB (especially at their rates) is more-or-less pure profit.

      I have a theory that AWS, etc capitalize on people not really understanding this anymore. AWS is 20 years old - that's an entire generation of CTO/CIOs on down that are completely unfamiliar with these details and think $0.10/GB or whatever is "just what bandwidth costs". It is not.

      [0] - https://bgp.he.net/AS13335#_peers

      1 reply →

    • I have heard that they rather drastically constrain QoS instead, which does sound reasonable. So you are still not charged for abusive traffic, but your service will be much slower than what is actually possible with paid tiers.

      5 replies →

    • I think a lot of people don't understand how cheap bandwidth is and is decreasing in cost practically every day. Amazon and Google have a lot of people fooled. Go ask someone operating in China and East Asia (and Japan) how much they're paying for local solutions.

    • it's 100% not sustainable. Use it while it's good, but don't get vendor locked in, because sooner or later they will increase the prices

      3 replies →

    • By the time it isnt sustaninable I will have IPO'd and be the next offensive new money tech billionaire writing threads on twitter telling you the secret to success is the 5am grindset and everyone who isnt sinking 5mil into the next big thing (tm) can have fun staying poor.

  • > Cloudflare's free tier is ridiculous: We do over 30TB+ of genuine traffic for $0

    It's not really ridiculous if you think about what you're giving them.

    You are massively benefiting their platform by providing them data which they use to train their services and then sell those services to other customers.

    I'd make a case that the data they collect is the most important part of their business and the free tier is a major component of this.

    • If you are not paying for it, you are not the customer; you're the product being sold.

  • I don't think it's fair to call it their free tier - it's their discretionary tier, there are numerous cases of the rug being pulled as and when it suits their business requirements to do so. Being left homeless vs. urgently coughing up is exactly the wrong problem to be dealing with mid-attack, I can't see any way to consider it free by any practical definition

  • I know that putting all eggs on one basket and giving it all to Cloudflare is not a good idea, if they have an outtage then I would also have it to. But when they are down, one third of the internet is down with them too. With 240$ a year for CDN, 60$ a year for serverless and $0.015 / GB-month for S3-compatible storage with free egress, I don't think anyone could find a better alternative than CF. I'm mixing with AWS, CF and self-hosted machines and the infra cost is less than 5k$ a year. Now I can spend the remain hard earned money for some fresh marlboro cigarettes.

Use a token bucket on your web server to catch abusive IPs and then blackhole them using `iptables -t raw -I PREROUTING -s ip -j DROP`. I know. I run https://ipv4.games/ which invites hackers to unleash their botnets, and the service runs on a small VM with only a few cores. It's been attacked by botnets with 49,131,669 IP addresses. There's no Cloudflare frontend or anything like that, because back when I used Cloudflare, the people who attacked the service would actually bring down the Cloudflare nodes before they brought down my web server. I doubt I've ever paid more than $100/month to operate the service. Please note that your service provider needs to have free ingress in order for this strategy to be effective.

  • This strategy may work for a (D)DoS that is targeted to an application layer, but won't work if the attack is designed to exhaust your bandwidth.

    Once you're receiving more traffic than you network cards can handle, it does not matter if you'll drop the packets with iptables or not.

    I was the target of attacks that caused Hetzner to terminate my contract. I was leasing physical servers there, so I assume the attacks were overwhelming their infrastructure.

    • I rent a GCE VM and there's not many if any people out there who can exhaust Google's network infrastructure. The only thing I have to worry about is making sure my server doesn't respond to abusive traffic.

  • Eventually you're probably going to want an ipset, at least. Otherwise processing your chain will continuously cost more, and more, and more.

    • I just declare firewall jubilee every now and then, where I flush the iptables and let people try again. It's also because people usually only control the IPs they use temporarily, so I don't want someone innocent later on to be blocked from using the service because someone abusive used their IPs beforehand. But even if I didn't do this, it doesn't cost much for Linux to iterate over an array of blocked int32's. It's really only allocated TCP connection resources that are problematic.

      1 reply →

If you want to sleep tight just get a dedicated server or VPS from something like Hetzner and/or combine with CDN providers like BunnyCDN - set up alerts just in case though. It takes more time and resources to manage it but you could save a lot on it in this case.

  • This so much. My hetzner (best choice for a media server within Europe) has 0 downtime in 1.5 years. And exactly as you said I am using bunny as well, which costs me a few $ per year.

  • > It takes more time and resources to manage it

    For most of the new web projects, setting up your brand new server is pretty well documented process and should not take more than couple of hours.

    It get complicated when you grow and add more servers or components. But at that point, you should be able to afford a part-time consultant to handle complicated tasks or just use Cloud then.

  • I'd even say build your system so as it can run on shared hosting. This way you even save the management.

  • That is my setup after leaving AWS for some of my services (low user amount b2b).

    I put in far less resources and maintenance after I had the system running. Especially if you need to manage the software running anyway.

This might be a good time to point out Cloudflare Pages: https://pages.cloudflare.com/

Under the free tier:

> Unlimited bandwidth

  • I'm moving everything to Cloudflare.

  • I didn't even know Cloudflare offered a JAMstack platform. I'm going to switch as I already use Cloudflare for domains.

  • Yeah, I'm already using Cloudflare because of Google Domains got de facto killed by Google via transferring it to Squarespace. Why not Cloudflare Pages, CDN, and R2 (S3-compatible storage) too? I'm even considering paying for the paid tier in the future if I ever go above the limits of 20 000 files per static site and the 25 MiB single file size limit [^1] (more than enough right now or in the near future).

    [^1]: https://developers.cloudflare.com/pages/platform/limits/

  • I was looking for a static site hosting option recently and tried out cloudflare pages. Fit my need perfectly. The generous free tier and the reasonable pricing model were the big factors.

    Oh, and the ability to put some authentication in front of it was a big feature for me.

Host on a provider which bills per hour. This caps your cost. It also makes your users pissed because you will go down, but if you’re small, you can afford that. If you’re big, you already have scaling options and should have a team to handle ddos.

  • My experience is that customers don't really care that much about small amounts of downtime no matter what size you are, people mostly get that unexpected stuff happens as long as you don't get hacked or misplace their data. Customers might complain a bit but seldom leave because of a few hours downtime.

    This seems to mostly hold true to developers also, GitHub manages to survive just fine after all.

    • Depends on your service. 20 second downtime on loading HN? Nobody cares. 20 second downtime on the last play of the Super Bowl - big problems.

      For most internet consumers we’re accustomed to poor service so if a page doesn’t load we’ll assume it’s a local problem and try again 20 seconds later, same with buffering, it’s just something that happens occasionally. This is increasing the case for phone calls too. Legacy live tv and radio going silent though is still a major issue, especially on live events.

      1 reply →

  • Yeah. Any host that won't infinitely scale out will solve this concern for you.

    • I think most people pick Netlify for it's Infra as a Service offering, so it would be nice if they had a way to throttle and budget in that offering.

      I would even imagine Netlify's target market is small to mid size businesses who really don't need ridiculous burstable scaling capacity at all. Seems like a bit of a trap door for that customer base.

      I agree though, I wouldn't host on them as a small business due to that risk, but I am also happy running my own server so I might be an edge case.

Vervel charges $400/TB for excess bandwidth, it's not even DDoS you should worry about, just moderate success.

  • That’s a crazy high bandwidth. Bandwidth isn’t free, but $400 will get you a month of 10gig in my local peering point, that’s 1TB in 15 minutes.

Imagine you lost your job. So you are here enjoying creating and hosting your hobby projects in theses services. Now, suddenly one fine morning you get slapped with $104K bill because someone decided to randomly ddos your one page dog lover website.

Now, who in the would would be thinking of having ddos protection for their hobby project? This is just absurd thinking.

  • No. This is absolutely common. I remember well how shared hosters 10 years ago already put caps on cheap packages and took the websites offline in case of traffic. And today it's Amazon who bills small players into dept.

    There are many provider who don't tho.

    • I always loved nearlyfreespeech.com for this, (prepay, and if you run out of money the site goes down) but found it to be a pain for projects that really needed a VPS

  • Can't hosts just make a site unavailable once it reaches its plan's bandwidth limit, DDoS or not?

    I think being offline is a lesser headache than a large bill, especially for those who are inclined to a free tier to begin with.

    • Folks regularly show up in HN comments during these discussions stating the opposite—that it's categorically better for all sites/projects, now matter how inconsequential, to stay online. It's weird.

      This includes some of the TPTB, too. Occasionally, though, someone'll say the quiet part out loud. E.g. re fly.io:

      > putting work into features specifically to minimize how much people spend seems like a good way to fail a company

      <https://news.ycombinator.com/item?id=24699292>

  • This may seem weird, but I believe ToS ae the real problem here. I call it the "car rental" problem.

    When I rent a car in person, I am often given a contract. And this contract is filled with tiny print, and pages of it.

    There are often people behind you, waiting, and bored/annoyed people behind the counter, waiting. This is beyond unreasonable.

    A point of sale contract should be short, in readable text, and understandable. For example, renting a car? Under a page, easily parseable, and if the person behind the counter cannot explain it, it is null and void.

    From a legal side, you can do this. And you can explain legal terms. Of course this means you are describing intent, which limits one in court, oh boo hoo Mr Lawyer. Cry me a river.

    Well the same should be true of any retail contract. Sign up for a service? One page with costs listed.

    At least then, there is hope of an end-user sort of understanding. And as one could claim that a DoS was actually targetting the provider, and not the website, that should be described too.

    So back to the topic at hand. I would write a demand letter, insistong Netify explain the charges, and ask them if they and their IP ranges were DoS, and if so that the charges be reversed.

    Because you shpuld not be paying, if someone attacks Netify.

    This letter should also be sent by mail, sig required, to the corporate address too.

    • > When I rent a car in person, I am often given a contract. And this contract is filled with tiny print, and pages of it.

      As someone who reads the agreements I sign, one thing that has become prevalent is that they're so used to people not paying attention to what they're signing that they're sometimes not even giving you an accurate copy to review. For example, you read the thing and think, "Okay, I can work within these parameters," then you sign, and later get an email containing your "agreement", but it turns out what's in the email is a different set of terms with a bunch of stuff that wasn't in the terms you actually agreed to when you signed. Or someone hands you a pad with an "I agree to the terms" box checked beside the signature line, and when you ask to see the terms you're agreeing to, they're caught off guard (being totally unequipped to let you do that), which turns into being flummoxed with how to proceed, which turns into getting angry with you for asking.

      1 reply →

    • Every rental and service is so optimized against scammers and abusers that being a perfect legit customer ie. simply want to pay, use the resource, then return the item or terminate the service, you're walking along the edge of a cliff. Annexes, penalties, fees and charges, exclusions, "sign this one more form, everyone signs it". Housing rental is another extreme example, one is simply unable to just get a job in new location and rent something long term.

    • This applies even offline! Have you ever tried to get a hold of exact insurance policy wording before going through their entire sales process? Impossible, in my experience, whether it's long-term insurance, vehicle insurance, pet insurance, etc.

  • It shouldn't be like this, but it is.

    Unfortunately, in today's world, DDoS protection is the equivalent of basic hygiene, foid and road safety. It's just a travesty that the hosting providers don't feel like it's their responsibility to address it.

“We leave your safe deposit box unlocked. You might want to forge your own lock and key. If we happen to notice someone stealing out of your box, we will let them grab as much as they can for one minute, then maybe install our own lock if our revenue is close to target.”

Can anyone share an example of edge middleware that might protect you on Vercel?

  • You can also turn on soft and hard spend limits on Vercel (including SMS alerts) https://vercel.com/blog/introducing-spend-management-realtim...

    • This is not a hard limit, this is just a webhook and you need to handle disabling everything on your own.

      1. You can make mistakes in your code.

      2. Some junior developer can by mistake change the code and make it ineffective.

      3. Webhook is not instant so you can get billed more than the limit.

      4. There is no information which project hit the limit, so you need to Fetch all of your project ID's and then disable all of them. You need to basically disable all the projects assigned to your team/organization.

      5. Vercel doesn't guarantee in any way that you won't get billed more, they are just sending an information to you (with a delay).

      Hard limit should be a deal between user and company that the user won't ever get billed more than X$

Those services exist, and you have the option to use them. Netlify is not one. Apparently, you chose that the un-insured solution was best for you.

Wait until you learn that Vercel only supports blocking IP CIDR ranges on the Enterprise plan.