Comment by dangoodmanUT
20 hours ago
Unfortunately for infra startups, I think we're going to see a convergence back to hyperscalers as a result of incidents like this and railway.
There's no amount of "nicer to use" that can replace security, at least certainly when selling to enterprises
I actually think it’s the opposite. I think people will move back to their own hardware with OSS deployment like dokku. If everything is more bespoke the attack surface is smaller as is the pay off. Hacking railway got access to lots of api keys, hacking some random vps gets less payoff.
> If everything is more bespoke the attack surface is smaller
This is exactly the problem that LLM assisted attacks solve. For a human attacker, the extra work to figure out a bespoke solution is too time consuming to be worth the effort. LLMs can search for vulnerabilities and spit out bespoke attacks like it's nothing. On top of this the bespoke solutions weakness leans heavily into the AI attackers advantage: response time. Even if they can just get your company's single Anthropic key, if they do it at 2am while your one infra person is on vacation, they have plenty of time to extract enough value for their effort.
I agree with the move to hyperscalers. The future (well, actually present) is going to be constantly seeing new attacks happening. The only real solution to this going to be having teams of people and agents constantly watching for these and able to deploy fixes and respond to risks immediately.
You’re technically right, which is the best type of right, but my point is the fragmentation and lower reward will make discovery harder, and the value of searching and cracking less worth the effort.
I had a html game I built for my niece on railway with an open ai API key so she could vibe code changes. No hacker would have looked for it but now I am catching strays because railway was a juicy target. Hyper scalers would be the same. Some random colo not so much.
1 reply →
> LLM assisted attacks solve
Yes, LLMs can easily knock over your on-prem docker appliances.
But you can still put stuff out of reach. I'm currently designing a key on-prem appliance that is can be compiled down to transistors, giving deterministic wire-speed performance while ruling out RCEs. It's cryptographically homomorphic: it processes authorized inputs but doesn't know what they mean, so an attacker who obtains the disk doesn't benefit either. The design will be highly resistant to exploits, and is likely to remain outside the "next-token" prediction training set.
This is the kind of thing AWS won't sit down and engineer for you. But thanks to LLMs I finally have the time.
How is that ever going to work out? The speed of discovery for LLMs is so fast, these places are getting breached more often than not due to a human errors. Its really hard for machines, let alone humans, to guard against a lot of LLMs going against them 24x7.
It's much easier to protect yourself if you have your infra behind a perimeter firewall you control.
17 replies →
No self-service sign up for your app. No public access to the app itself. Website that let's you submit a sign up request form. Team reaches out and reviews your request, creates an account in the app dash for you, and you set up mTLS to get access to the dash.
20 replies →
For one, I have strict iptable rules to drop inbound from outside my specific commercial subnet and my specific residential IP. After that, all the usual best practices.
Breaking a lock still costs approximately the same, whether the lock is protecting a small barn or a huge hangar. A large number of small but reasonably tough (not unbreakable) targets makes the payoff per target smaller. This necessitates target selection, and chances of being deliberately skipped grow for an average target.
2 replies →
This is absolutely happening. I'm not a developer in SV, I'm an IT leader out on customer land, and I can tell you a LOT of companies are looking to bring things back on prem for security and cost reasons. And before anyone laughs, yes, you ABSOLUTELY can save money going on prem, I've done it for company after company. It's one reason the PE firm that I've worked under for a decade kept moving me to new companies.
Let them laugh. Self hosting is much easier with OSS that solves most of the pain that made the cloud transition worth it, and commodity hardware has been cheap and capable for over a decade.
1 reply →
My company just did this. Still using AWS S3 but everything else is on-prem with kubernetes.
4 replies →
This is misguided thinking. You are less likely to hear about on-prem breaches in the news just because they are less likely to be newsworthy incidents, but they are even more likely to happen. Part of the reason people moved to the cloud is because it is much easier to have a team manage updates and security at scale.
Being a small target is not helpful these days, attacks on on-prem software today simply scans the internet, or uses databases by people who have already scanned it, and is automated.
On-prem software has not been spared from the recent spat of security incidents.
There is a difference of the security of software running in your cloud resources rented from AWS etc. managed by you and being a tenant in many SaaS systems all cobbled together all running on rented hardware. There is a lot more attack surface and a lot more diffusion of responsibility.
My comment[0] tries to explain the nuance and I would agree with you and both the GP in some sense.
For Entreprise, the shift will be towards hyperscalers as they want trust and less blame (Nobody got fired for picking AWS)
For consumers, we care about pricing and stability and control and we will move back to our own hardware or cheap VPS providers and self-host things.
The middle ground would be where the slight pain could be at in my opinion.
[0]: https://news.ycombinator.com/item?id=50307410
> For consumers <snip> self-host things.
That's not how consumers work. That's a nerd/geek/hobbyist mentality.
If Ikea and Wayfair went out of business you wouldn't expect consumers too take up woodworking and build their own furniture en masse.
For consumers/small orgs, how is the security at a cheap VPS provider going to be any better than at a cheap IaaS provider?
4 replies →
It's probably both.
Cost of hacking a custom setup is going down rapidly. There was broad agreement that the PHP vulnerability was very difficult to exploit in practice. It took two hours for someone to get into Steal My Bitcoin. Apparently all they did was tell a decent model "go exploit that." AI isn't free but it's not that expensive.
If you have to hack 1,000,000 custom setups to get the same results as one hyperscaler, would you bother?
1 reply →
No thats an old way of thinking because it assumes mass targets are better because hacking is a high effort activity. But with the automation of intellect the unpatched systems (which a much more numerous when everyone self hosts) is the key point.
Mass credential stealing is just the top funnel of hacking and (compatativly) isnt that lucrative - these credentials are then sold on the black markets to the next step of hackers who use them to exploit targets.
With unpatched systems and self hosting, this second group can just set an llm at a target and have a much higher chance of success without needing credentials
It would be a dream for the infiltration hackers to have everyone self host. Unpatched and miscondigured systems would be everywhere and LLM automation could make mass scanning them and exploiting them very attractive - not mass scanning and exploiting them for credentials stealing, no! For layer 2 stuff
> There's no amount of "nicer to use" that can replace security, at least certainly when selling to enterprises
Azure would beg to differ.
https://news.ycombinator.com/item?id=50301854
None of my customers / clients / coworkers have ever used Azure and "nicer to use" in the same sentence.
and it wasn't what he was saying, the "nicer to use" in the context were startup IaaS alternatives
A moron could build a better UX and DevX.
I have a bridge to sell you if you think everyone on the Azure team is sub-moron. Infra security is a hard problem to solve, GCP and AWS don't quite have a perfect track record either, perhaps AWS stands out as a better option but it's hard to argue about it.
15 replies →
Are hyperscalers actually more secure though? I have only heard anecdotal thinking eg nobody ever got fired for choosing IBM.
It’s the same thing: let’s say Microsoft or AWS got hacked. Then you’re definitely not the only one who got hacked.
So, you can always say: Look, it’s not us (especially not my org!). It f’ing Microsoft/AWS! Everyone got hacked.
You and your career are gonna be just fine.
In a prior role, we realized something similar with outages. For our particular product, we AWS went down - we went down. We freaked out the first time, but after it was done we realized not a single customer had inquired. This felt unusual since we were getting emails quickly if we have a self-inflicted outage.
Turns out, when a major provider went down all of our users were too busy fighting their own fires. When they do notice, it's on a day when ALL of their vendors are struggling so we don't get singled out as a bad apple.
The whole "No one gets fired for using IBM" analogy.
This is especially true if the majority of your customers are also on Microsoft/AWS/etc.
Literally 0 of your customers will notice that you are down if their own systems are also not working.
This is perfectly valid for many businesses, up to a (shockingly high) level.
2 replies →
Yeah, if Exchange goes down, you get a day off. If the little boutique email provider you chose goes offline, you get a day in calls with everyone yelling at you.
It's today's "nobody ever got fired for buying IBM". You're really just want the the SLA (someone to blame when things inevitably fail)
If you need deniability, sure.
But it you actually NEED security, it's clearly not an option.
4 replies →
You are just describing how they're a more valuable target for an attack.
With a lot of PaaS type vendors you are effectively exposed to both the vendor themselves and the public cloud they run their stack on top of.
Broadly I think you're much more likely to be the victim of a platform-level breach involving an immature startup PaaS than a hyperscaler cloud, however I think for small teams the alternate risk you take on is that AWS/GCP/Azure are full of landmines and extremely easy to misconfigure in insecure ways if you don't know what you're doing. Especially if you have AI telling you confidently wrong things.
When I was at Microsoft we took security cripplingly seriously to the point we could not deliver features and our competitors could because the security process was convoluted and had multiple reviews and approval steps.
Having crippling bureaucracy does not correlate with good security outcomes, except maybe negatively. However, having a history of unending egregious security failures, like Microsoft, does correlate with poor security outcomes. For that matter, having a present of unending egregious security failures also correlates with poor security outcomes.
Let us look on the front page and see if there are any, oh look here is one: https://www.binarysecurity.no/posts/2026/10/one-root-case
7 replies →
Practically every software security firm in the country was a vendor to Microsoft in the mid-late oughts and I know the Microsoft you're speaking about, but the vibe I've got lately is that things are different there now.
Tell me you weren’t on the Windows team without telling me you weren’t on the windows team :)
Unquestionably. They have top tier security operations that are well funded. They follow (or set) security best practices, they are audited, and they fix them quicker than you can. They have inside industry knowledge that is only shared among top organizations in this space, including government cyber ops. This gives them the ability to fix things before they are even publicly known.
You can get close if you have either:
1. the ability to spend hundreds of millions on security operations yourself
or 2. the ability to airgap the systems you are securing
> nobody ever got fired
thats the crux; are you working for the paycheck, or for the truth?
could be both, but pretty much all "i was right" stories start with "everyone else said i was wrong"
For me, the truth. I’d like the most secure provider possible. Which one is it though?
1 reply →
[dead]
Yes and no. They (or anybody else) are not guaranteed to be secure, of course; see all the compromises regularly found in cloud providers; Azure especially has had so many cross-tenant vulnerabilities (there’s one on the front page right now!) that I’m amazed the concept of public multi-tenant clouds still exists.
But they are also very highly incentivized to be more secure. They have millions of customers (hence a correspondingly higher expectation of trust) and billions in revenue from the trillions in economic activity happening on those platforms (hence a correspondingly larger target on their backs.) This means exponentially higher liabilities. And that probably matters more than any other factor.
Adding on to it, even though they may be using AI as well. This could very well be a placebo and deniability factor but the fact remains that they have many many qualified employees and the chances of bugs passing through that are much less compared to other systems. Also it's that the systems are very severely pen-tested and have years of bugs removed out and the incentive of billions of dollars at stake that you mention that they have for the trust premium.
They will still make mistakes but they have the resources to make them less.
I doubt that such intense level of thorough testing can occur in a smaller company.
I still love small VPS providers and use them myself but it is worth knowing why enterprise is willing to pay a premium over those to these hyperscalers (it is a trust premium)
The things that Northflank and Railway were apparently breached to steal aren't just things that hyperscalers would have protected worse instead of better; they're AI API keys, things that only existed because people were using hyperscalers instead of running inference on their own hardware (or on Amazon AWS EC2 GPU instances).
People have been breaking in to hosting companies to steal bulk secrets since long before there were inference tokens to steal; the modal attack probably targets database DSNs.
[flagged]
2 replies →
Why do we think hyperscalars are likely to be immune to this?
They aren't immune, but they almost universally have better security posture than any alternatives.
I hope this is true! But I don't think I'd make a bet that we won't have a big breach like this of a hyperscalar in the next 6 months...
Theres always a market for seedbox providers. Hyperscalers will always wear the white gloves and enable this market.
What about Railway? Some big incident happened?
https://blog.railway.com/p/preliminary-report-october-securi...
Oh boy.
“malicious actor had access to abliterated models and substantial compute resources which enabled them to perform highly sophisticated exploits against our infrastructure.”
They don’t provide any details but pass it off as an advanced AI attack. I’m going to just read that as what it is. A likely scapegoat for shitty security.
It's also going to be the downfall of open models.
If you think governments (US and China) are going to tolerate people using models for anti-social behavior, I have a datacenter to sell you.
This will be the downfall of the torrent protocol. If you think governments (US and China) are going to tolerate people downloading copyrighted games, movies and music without paying the rightsholder, I have a datacenter to sell you.
Pirated content can run on a $20 computer.
SOTA open models need tens of thousands of dollars worth of hardware to run.
I get your analogy, and it feels right, but the compute requirements to run these models are extreme. That analogy falls apart when it necessitates having a corporate 3rd party run the models for you. They will not tolerate breaking laws.
If China says no more, then the best the masses are gonna get on their high-end setups are the 30B class models. Which are decent, but not "hack my bank", errr, "protect our bank" decent.
Piracy flourished because you only need an operating laptop to participate. LLMs you need a server rack of GPUs to participate.
1 reply →
azure just had a huge leak
IMO, There is no "middle-ground" to the situation.
I wanted to write a blog post about it as I was thinking the same but I guess a comment will help me get my thoughts across just as much.
Either we are gonna see a convergence to hyperscalers as you mention, or we would go back to completely indie companies which compete on perf/price.
There could still be some value to "nicer to use" certainly but with AI, we are gonna see almost every product as being "good-enough" if I am able to convey my sentiment.
An anecdote is: Proton's an middle-ground company. I am seeing the Verge article which states that their CEO has gone a little crazy over the use of AI and the wording makes me lose trust within it. Proton's whole point is security and yet it's having this, It has some stocks in non profit and is a swiss company. It shouldn't have been this way and would've been one of my least anticipated companies but I guess they want growth at all costs and the growth has been slightly lesser than expected and now they are burning the good-will in many ways.
I have to wonder that even if proton might have good engineers, then within this pressure by their CEO/CTO. I cannot trust the engineering talent because of these issues because they are very much forced by their company to use AI.
Proton is also really expensive.
Now just today, also, with a company so big as binance getting hacked and losing 6.5 Billion$. My trust within security just because its a large company is less.
On the other hand, I know a guy called Vikas on lowendtalk who operates his own mail platform called mailbig. I know he vibe-coded the landing page and might've vibe-coded something but it is really cheap. I also know francisco whom I have talked to who runs namecrane. (I think mailbig uses namecrane in many ways)
Perhaps E-mail can be a sensitive matter because you have to place trust in the provider even more because the protocol is not really that much secure but suppose it's an matrix instance (where it is genuinely encrypted) then that shifts it even further. I do trust NameCrane and NameBig from what I can gather.
I'd much rather support them and their cheaper deals than some other middle-corp. Sure some security factor can come in but also, the difference feels comparatively less and the balance has been made
The market will slightly squeeze as Hyperscalers will introduce the solutions of the middle-ground and the new players will everyday enter into the field with more competitive pricing just as well.
So in short, Enterprise will move towards the hyperscalers. Generic Consumers will move towards the cheapest option. The moat of the middle ground where UI/UX was the differentiator is being turned down. There would be some mix and match and there will be cost-savings for Enterprise to move from hyperscalers but it would depend on a lot of factors.
I have news for you. A lot of enterprises are moving a lot of workloads back on prem and are expanding their datacenters and on prem GPU compute. Also replacing a lot of SaaS with internal tooling.
If you look at the numbers for revenue at the hyperscalers right now this move doesnt seem significant.
It is absolutely cutting into their growth. And EU migration off is also a hot area.
2 replies →
They are also getting hacked, and they are much jucier targets.
>There's no amount of "nicer to use" that can replace security, at least certainly when selling to enterprises
We can lower our standards instead
[dead]