Comment by arjie
21 hours ago
The biggest mistake I made was high uptime. arjie.com was up for 10 years plus on a Hetzner VPS so that by the time they wanted to sunset the machine underlying I had no idea what my teenage self had set up. I have the backups but the site hasn’t been up in a decade…
Nowadays I build things so that they move and I have moved things about a bit so I know they work.
"The biggest mistake I made was high uptime"
Quite. I'm old enough to remember machine uptime being a badge of honour.
However, being older and not really wiser, I look for service uptime these days. Yes we did have similar back in the day, that's why MX and the like DNS records exist.
Old school clusters were pretty esoteric but the lessons were learned (split brain n that) and that's why we still argue the toss with kiddies about why a Proxmox cluster with two nodes is fucked and why we recommend an additional "witness".
I don't care that VMware glossed over the whole two node HA cluster thing years ago with a massive bodge. They were wrong then and they are probably still wrong because that nonsense is probably still baked in.
Sorry, slight digression.
High uptime implies no patching. We all love patching.
https://en.wikipedia.org/wiki/Split-brain_(computing)
The more you know!
>a Proxmox cluster with two nodes is fucked and why we recommend an additional "witness".
Reminds me of the three Magi from Evangelion: https://magi.kinta.ma/
"a man with two watches can never be sure as to the time"
need a third one to confirm which of the 2 is accurate
There is something like live patching.
One reason mainframes and micros are still around us, is that you can change almost everything between hardware and software without downtime.
It is also available in commercial surviving UNIXes, and as paid for feature in some Linux distros, although not to the extent that those grandparent systems are capable of.
The problem with live patching is twofold.
First, you might not reload everything in memory, so it will be patched on disk but not in process.
Second, you have not tested that the system can boot to a functional system. Say you have done live patching for 5 years and never rebooted, and then you have a power loss or hardware failure/upgrade that takes the system down. When you try to bring it back up, it doesn't work. Which configuration change in the past 5 years caused that? Which backup do you use?
And, yeah, everything is hot swappable on VAX. Those machines also cost 6+ figures, and often require a service contract that includes a permanent on site tech.
8 replies →
A Danish bank found out that this can bite you in the ass.
When you hotpatch the system for years then you have no idea if the system can boot up or it will fail somewhere in the booting process.
i.e. you can only trust what you regularly test.
5 replies →
You should't need mainframe for 100% (or five nines if that's fine) service uptime.
You can build that way cheaper with 2-3 proper clustered load balancer units, 2-3 application servers behind those and those using persistent storage (databases,ldap, files) which allow writing multiple nodes simultaneously.
I used to work uni that we had few services from 2012 to 2025 my retirement with zero downtime. One time my manager with tech background tried to add PBR in hurry using WebUI and did not understand cli syntax and caused close to require reboot, but I was able to fix it from cli rolling back previous config and rebooting one unit at time. Upgrading software major version up to each unit supported level wasn't hard, upgrade node it joins back cluster, upgrade another node and it joins cluster, all done. Few times I had to fix manually config for some less important test backend servers that I had forgotten to change before upgrade. No big deal. No major outages during all that 13 years time happened. Some redirecting policy and action syntax was first hard to understand and learn like GeoIP, but I was very surprised how darn reliable and nice they to use and maintain.
The LB's were (Citrix) Netscalers in clustering mode (all nodes process traffic concurrently), which allowed live update one node at time without losing any connectivity through them. That wouldn't have been possible devices in just HA mode.
We had just 2 beefy units which worked very well for us, but you can have 2-32 of them in cluster and managing thousands of servers behind them if you need that. Netscalers are FreeBSD derived where quite a bit of the TCP/IP stack was rewritten adding support many some quite odd features std FreeBSD doesn't have. Much of that is IP/ethernet multicast features, PBR's, Traffic Domains (VRF's) and of many service and monitoring processes which sync cluster (or HA) and if node fails another can continue straight from there without any loss of traffic to clients being proxied.
Though I think most people in this forum are familiar with with haproxy, pound and web-server software provided reverse proxying.
A car analogy if previous were your fancy sport sedan Netscaler and F5 BigIP are formula F1 class cars ie. quite different beasts altogether.
e: And proper LB's are not just for HTTPS etc. but very nice proxying many other protocols were they TCP, UDP or something else. We did done VPN's and something like Cisco AP'S CAPWAP (DTLS ie SSL over UDP). e: typo.
1 reply →
I’ve long wanted that amazing uptime and virtualization and huge I/O and all that cool stuff mainframes offered, but on the desktop or in the closet, with modern CPUs.
I think I’m gonna hafta keep waiting...
> One reason mainframes and micros are still around us, is that you can change almost everything between hardware and software without downtime.
We have some Sun V880s at work and I'm fairly sure the only part you cannot change with the power on and system running is the motherboard itself.
And I would not be surprised if some ex-Sun Gandalf Beard "well akshully"s this comment.
1 reply →
My raspberry pi serves only to be the tiebreaker my possible split brain 2 node cluster lol. It is literally called tiebreaker
In 2012 I took over a Perl project that was running on 25 BSD servers (OpenBSD I think?) that had not been updated / patched since 2000. It was an interesting time.
> Yes we did have similar back in the day, that's why MX and the like DNS records exist.
Care to elaborate? I wanna know more.
MX records publish an SMTP server for a domain and a 'priority'. You can have multiple MX records and (theoretically[1]) you try the one with the lowest priority, and if it doesn't respond, try the next lowest, etc. Or (theoretically[1]) if you have 2 MX records with the same priority, you can load balance between them.
https://www.cloudflare.com/learning/dns/dns-records/dns-mx-r...
[1] yes...I know there's a ton of caveats here...
two is the right minimum number for a high availability dataplane but three is the right minimum number for a HA control plane.
With that said, if high availability is not a concern then 1 can be just fine.
It's pretty easy to abstract away a proxmox node into a terraform or other type of code based recipe for easy backup / reconstruction / upgrading.
This reminds me of Ise Shrine in Japan, which is completely dismantled then rebuilt every 20 years.
This is top of mind because I recently read Breakneck by Dan Wang. He makes the case that this practice of rebuilding the shrine preserves knowledge that would otherwise have been lost to time. Wang contrasts Ise Shrine with Notre Dame, where rebuilding the roof is apparently quite difficult, perhaps in part due to the loss of knowledge. I'm not familiar enough with either structure to judge whether this is a fair comparison, but I like the principle.
(Edit to add: This is only a minor analogy from the book, which I highly recommend overall.)
Thank you for the recommendation! I love that reference, and particularly because I am fond of the story of the shrine for a different reason https://wiki.roshangeorge.dev/w/Constancy_Preference#Concept...
Indeed, for a VM, high uptime makes little sense, because a reboot takes a few seconds, and an upgrade requires no downtime, just switching the DNS to a new instance.
For a physical machine which you can't easily copy, it's a different story.
We're talking what, 15 minutes to reach post?
I started putting things in a big ansible playbook repo. Don't need to have it fully managed by ansible either I mostly just have setup configured there I still do lots of by hand management.
I have the same. The infra management is in one place, the apps hold their own, and there’s a docs folder on the server where each guy puts his stuff. The install is idempotent deploy scripts. But back then my stuff was more ramshackle.
Sometimes I leave Architectural Decision Records for personal projects. It feels silly but it honestly comes in handy more times than expected
I keep them embedded in the codebase or an artifact right next to the source.
And the key thing is that i dont need too many details at all. A few cues and its all back in my head.
I hear you. On the other hand, not having to mess with something is good. I just make extensive notes in a README somewhere - usually in KeePass right next to the system info.
I stood up a dokuwiki instance recently and then documented how to stand up dokuwiki, haha.
I disabled revision history viewing and have a public portion and a private portion. I use it to track things I'm learning and document rollout procedures and commands I need for things. So far I have rclone backups into S3 Glacier, Tuwunel(Matrix) server deployment with voice/video support, and various little tutorials on server stuff I'm learning.
TLDR use a wiki!
> The biggest mistake I made was high uptime. arjie.com was up for 10 years plus on a Hetzner VPS so that by the time they wanted to sunset the machine underlying I had no idea what my teenage self had set up. I have the backups but the site hasn’t been up in a decade
LLMs have solved this problem, they’ll happily deal with the software archaeology on your behalf. This is the kind of task they really excel at.
You're right, of course. At this point it's inertia. It's been dead a decade.