Comment by bithavoc

More like the rack isn't viable for SaaS.

I believe JPL and INL were their launch customers for gov.

The rack were not insanely expensive, expandable so you don't have to build fully stacked racks from the beginning. Purely based on a hardware, compares to other blade systems but pricier.

The interesting part was the software, management interface, Terraform provider and how everything just fit together. Having storage, network and compute all in a single managed rack package brings a lot of value and brings down TCO. Really appreciated the security group like approach to network policies.

Unfortunately I haven't worked with HPE or Dell recently so I'm not sure what they currently offer.

You don't go to oxide for lowest cost.

How so?

It is worth the exercise to honestly assess the maintenance, initial costs, ongoing costs, etc etc for owning your own. I did this at a company for a single server to let the ml group test fine tuning frameworks/throw crazy ideas at before spending real money on a training run and it paid for itself in like 3 months. Sometimes the math is overwhelmingly positive, sometimes it isn't. In general, the smaller/earlier you are the less it makes sense. Not because of the per hour costs, but because of the distraction away from implementing the core idea of the company. It is worth some, maybe a lot of, inefficiency to stay focused. When you get bigger and have strong fit and a clear direction it is a lot easier to focus on optimizing hosting costs and spending leadership time away from the core problem of the company.

Oh man I cannot tell you how many times I've gotten into arguments on here and other technical forums years ago about self-hosting and how 'stupid' I was to not want to rent cloud servers

You’re expressing a very narrow engineer’s perspective that doesn’t consider the realities of managing bare metal hosting at any kind of scale.

These are business decisions, made in terms of core competencies, capex vs. opex, and the difficulties and cost of building out a reliable, sustainable hosting operation that handles all the compliance and security requirements, and the full range of “ilities” that real businesses have to deal with.

The fact that Bob in IT might be capable of doing some of this on his own doesn’t really enter into the picture. It’s not relevant.

There’s a reason that most companies don’t operate their own electricity generation systems. Much the same is true for computing systems.

That is the starting point though , minimum you need to buy one more for FO/DR if you care about uptime .

It is not once and done you have some continuous costs in either direction.

1. you will not be able to still move all your workloads only your base loads .

2. There will be residual workloads not only if they are spot/flexible bursts but also they are too deeply integrated to Paas offerings .

3. you have to pay for power cooling and network (and its backup) for 5 years .

4. Finally you also have pay for infra team to maintain, this is increasingly difficult (and expensive) to hire and retain .

Don’t get me wrong I would love to be able buy something like this, but the cost economy is pretty steep and 1-2M spends is too early (it is not on oxide, serious hardware costs a ton these days )

Which many keep forgetting is that there was no one else wanting to acquire Sun, IBM did an offer that was shortly thereafter withdrawn, and that was it.

Sun would have died, everything completely lost among creditors and that would be it, end of story.

How about the sellout of Joyent to Samsung?

VC funding is not for "a sustainable long-term business".

It makes sense for Broadcom to remove that avenue of escape, and it makes sense for Oxide's investors to charge a premium to Broadcom, and materialize their returns.

The customers will have to deal with it, of course. At least they bought physical systems instead of renting them, so they can use them until they're obsolete.

Makes sense for Broadcom.

Managing a single commodity device is pretty simple. Could even be a shell script with a few IPMI commands.

Managing 100s of devices from a mix of vendors is possible becomes itself a massive ball of crap to stick all the pieces together.

Then add firmware management because now there is a plethora of firmware updates to worry about.

And then dealing with networking discovery & orchestration for such setups is an added horror few even try to visualize.

Then abstract all these differences away so one can provision two different vendor’s servers connected to two different vendor’s managed switches in different racks to a new VLAN. And do it with SR-IOV while you’re at it…

Oh, I forgot shared storage… details, details…

Commodity computing is the modern day Tower of Babel.

Yes, it can be done much simpler if one treats all switches as unmanaged, all storage as local, and all networking as flat. That’s just not acceptable for a lot of use cases though…

Good question. Can’t eliminate the BIOS/UEFI on commodity systems. Can’t remove/cut down the BMC. Can’t plumb the hardware root of trust into the OS and VMs. Can’t do dynamic rack-level power capping (future roadmap). There’s a whole bunch of the software stack that only works because the hardware and software were designed together.

Oxide exists in part because commodity systems don’t work for building a cloud at scale (Joyent). Similarly, AWS, GCP, and Azure don’t use commodity systems, they use hardware that was designed to work together with their software, Nitro being a prime example.

> Is there anything they're doing that can't be replicated by hypervisor/management software on commodity x86 servers? Seems like their "secret sauce" is a software stack that "just works" more than any actual special feature of the hardware.

There's a reason Apple is consistently one of the top 5 most valuable companies. It isn't because their hardware does anything that can't be done on "commodity" hardware, its that they built a software stack that "just works" and part of that was tight coupling to the hardware.

It's not actually secret — it's open source! We talk openly about our architecture.

Bryan has a good take on the incentive structures holding back commodity hardware vendors: https://m.youtube.com/shorts/O8GSWKpK79s

Their machines are x86 but they've rearchitected basically everything else in the pursuit of a cohesive and integrated machine, they don't even have a BIOS. The secret sauce is having full control of the software stack, and as close to full control of the firmware as is possible on x86.

What can't be replicated is a culture of simplicity, quality, and security. I've seen this firsthand working at IBM.

(Former oxide)

Yes, custom hardware is a significant part of Oxide. You have to build your own to do that stuff, and that’s why they did. I alluded to some of the things upthread.

I found out about this round from this thread, just like everybody else, but

> Oxide having to raise money multiple times might be a hint

That’s not the only reason to raise a round, by far, especially when you, you know, are building custom hardware. It isn’t a SaaS business.

EDIT: here’s another commentor with an example of this: https://news.ycombinator.com/item?id=49176704

Complete hardware and software and firmware BOM. You won't get that anywhere.

> Broadcom with its VMWare acquisition could easily take these guys out if they wanted to.

Depends on what you mean by that. Broadcom cremated a lot of VMware's goodwill in the market.

> Is there anything they're doing that can't be replicated by hypervisor/management software on commodity x86 servers?

It was technically possible with (e.g.) OpenStack for years (decades?) before Oxide ever existing, and yet even with such a solution being around, some folks still went with Oxide. (Or, depending on the scale you want to talk about: Proxmox, XCP-ng.)

(Former Oxide)

An in-house team is most likely competing with something like Dell or VMware, not really Oxide. A significant part of Oxide’s value proposition is that you’re buying hardware and software purpose built for each other. Unless you’re also going to go so far as to do all of that, which companies like Google do, of course, it’s not really the same thing.

This matters when your various vendors start pointing at each other when something goes wrong. Oxide is truly “one throat to choke” in a way others just aren’t, and stand by that quality.

(Not to mention other various efficiencies, like power, or removing things like the BIOS and BMC junk that’s in basically every other server you buy right now. And the ability to send attenuation from boot up through the host OS. Just tons of things they’re differentiated on that your in house team just isn’t going to do.)

VC or private equity fueled companies are weird. At my place we are spending 8 figures a year just in AWS, and it's not like they planning to move to bare metal but, they're in fact removing stuff from their old datacenter. And that bill doesn't include some of the other SaaS like Mongo or Elastic.

With bills of that magnitude, each time I do a little house cleaning and delete some old data, change storage classes, or discover some unused servers... the savings (that are barely a rounding error on their bill) could pay for a whole year of an engineer or a bunch of servers that could power a good chunk of their production traffic.

Its not about bare metal, its about owning the whole stack and giving visibility into every nook and corner, and then supporting that open sourced code. Every network switch, every interconnect, every piece or firmware or software they can attest to because they built it themselves, and that is worth its weight in gold to high criticality industries and governments. They can give you the provenance for every single hardware and software component. Every chip.

If you can roll around those possibilities in your head, you start to see how absolutely unique this is and why these extremely sensitive and important customers and industries want this.