The biggest mistake I made was high uptime. arjie.com was up for 10 years plus on a Hetzner VPS so that by the time they wanted to sunset the machine underlying I had no idea what my teenage self had set up. I have the backups but the site hasn’t been up in a decade…
Nowadays I build things so that they move and I have moved things about a bit so I know they work.
Quite. I'm old enough to remember machine uptime being a badge of honour.
However, being older and not really wiser, I look for service uptime these days. Yes we did have similar back in the day, that's why MX and the like DNS records exist.
Old school clusters were pretty esoteric but the lessons were learned (split brain n that) and that's why we still argue the toss with kiddies about why a Proxmox cluster with two nodes is fucked and why we recommend an additional "witness".
I don't care that VMware glossed over the whole two node HA cluster thing years ago with a massive bodge. They were wrong then and they are probably still wrong because that nonsense is probably still baked in.
Sorry, slight digression.
High uptime implies no patching. We all love patching.
One reason mainframes and micros are still around us, is that you can change almost everything between hardware and software without downtime.
It is also available in commercial surviving UNIXes, and as paid for feature in some Linux distros, although not to the extent that those grandparent systems are capable of.
In 2012 I took over a Perl project that was running on 25 BSD servers (OpenBSD I think?) that had not been updated / patched since 2000. It was an interesting time.
This reminds me of Ise Shrine in Japan, which is completely dismantled then rebuilt every 20 years.
This is top of mind because I recently read Breakneck by Dan Wang. He makes the case that this practice of rebuilding the shrine preserves knowledge that would otherwise have been lost to time. Wang contrasts Ise Shrine with Notre Dame, where rebuilding the roof is apparently quite difficult, perhaps in part due to the loss of knowledge. I'm not familiar enough with either structure to judge whether this is a fair comparison, but I like the principle.
(Edit to add: This is only a minor analogy from the book, which I highly recommend overall.)
Indeed, for a VM, high uptime makes little sense, because a reboot takes a few seconds, and an upgrade requires no downtime, just switching the DNS to a new instance.
For a physical machine which you can't easily copy, it's a different story.
I started putting things in a big ansible playbook repo. Don't need to have it fully managed by ansible either I mostly just have setup configured there I still do lots of by hand management.
I have the same. The infra management is in one place, the apps hold their own, and there’s a docs folder on the server where each guy puts his stuff. The install is idempotent deploy scripts. But back then my stuff was more ramshackle.
I hear you. On the other hand, not having to mess with something is good. I just make extensive notes in a README somewhere - usually in KeePass right next to the system info.
I stood up a dokuwiki instance recently and then documented how to stand up dokuwiki, haha.
I disabled revision history viewing and have a public portion and a private portion. I use it to track things I'm learning and document rollout procedures and commands I need for things. So far I have rclone backups into S3 Glacier, Tuwunel(Matrix) server deployment with voice/video support, and various little tutorials on server stuff I'm learning.
> The biggest mistake I made was high uptime. arjie.com was up for 10 years plus on a Hetzner VPS so that by the time they wanted to sunset the machine underlying I had no idea what my teenage self had set up. I have the backups but the site hasn’t been up in a decade
LLMs have solved this problem, they’ll happily deal with the software archaeology on your behalf. This is the kind of task they really excel at.
Personally, I've been running with Caddy in front of Docker (compose) for most of my personal/hobby usage. If it's a straight website, I'll let Caddy serve the contents directly... for "web apps" I'll pretty much containerize all the things and use caddy for TLS termination and reverse-proxy duties to the app running under Docker...
Mostly ~/apps/appname, where each appname has a docker compose file, and the data directories mounted under appname... I can compose down and (s)ftp the data out for hard archives or to move a site/service. I had been running a few VMs under a dedicated server, but switched to separate VPSes on OVH. Only gotcha with OVH is if you want to run mail, you want to avoid the local zone VMs that don't allow mail hosting.
I have started using Traefik on one of my projects and it's a nice upgrade from nginx proxy manager. NPM is great, its web gui is noce, but with Traefik all I need to do is write what I want to happen in the docker compose file and that's it.
My home setup is kind of this but it's unraid running the containers and one of them is an nginx thingy specifically designed for reverse proxying other services.
This is more or less how I do it, too. Debating switching from Debian to something like Flatcar which is container-first with an immutable system/OS.
I think FreeBSD has some interesting technical merits, but like it or not Docker is the default for a lot of open source software, and I have neither the time nor the inclination to translate everything to freebsd jails.
I've had my private servers running Arch (managed by Ansible) for the last 5 years but have recently been looking into Talos for the same reasons. Setting up a single node k8s using Talos was actually pretty straight forward. In the end I decided against it just because I couldn't justify the continuous load caused by k8s components when my stuff is only sporadically in use... Instead I now have them on NixOS and I really enjoy the declarative approach (although the language annoys me).
Oh wow. I did the same thing a couple of weeks ago. The server hadn't been updated since ~2015, running a blog on Ghost from that time with node 0.10 installed.
I was a bit rougher though: I just took a backup, then let my Hermes agent (Gemini 3.1 Pro) loose on it. It upgraded everything that needed to be upgraded, patched what needed to be patched, then proceeded to migrate everything to it's most recent equivalents. After that, a fair bit of server hardening was carried out followed by debloating of unused services. Likely would have continued to procrastinate doing this if it wasn't for AI support.
I've been using ZFS for about a decade on several systems and can't say enough good things about it: rock solid, feature rich and easy to use are the top benefits.
And it's been used by many commercial networking and storage appliances too. Juniper using various devices (routers, firewalls, switches, etc), Citrix Netscreen load balancers, Dell storage Isilon just to name few. Knowing FreeBSD pays dividends if you work enterprise gear. Makes your life so much easier even there is vendor specific UI's and shell, but you know your way around so much better when you encounter any issues which requires using shell and it helps you debugging things you would otherwise not being able to accomplish.
Knowing you way around *BSD how check things, mount a USD drive, collect data & or evidence when need arises is well worth having beyond just surviving bare Linux skills. BSD's are alive and kicking on commercial appliances and devices.
I enjoyed my foray into trying FreeBSD for my personal server. There's something cool, clean, simple and "punk rock" about it. But I gave up as my main pain points were:
- PM2 was buggy on FreeBSD, which I used to manage my processes
- An alternative, using `rc.d` to run daemons was just so hard to get logs working.
- The firewall required too much self configuration to get it right with all the best security practices (ie. What does one do with ICMP.) I was missing something like a template with the defaults that come with UFW, for instance.
For a very easy single-machine firewall, one could set `firewall_type=client` or `firewall_type=workstation` if you want to host anything. For the latter, `firewall_myservices` and `firewall_allowservices` control what ports are enabled and who (other networks/IPs) have access to them.
For a very simple NAT gateway, one could set `firewall_type=simple` and then `firewall_simple_(iif|inet|oif|onet)(_ipv6)?` to configure the ISP-side and internal-side interface names and IPv4 and IPv6 network ranges for each.
> - PM2 was buggy on FreeBSD, which I used to manage my processes
For supervision?
> - An alternative, using `rc.d` to run daemons was just so hard to get logs working.
The unix way is to use logger(1) If you only want some simple message, or redirect to files using newsyslog(8) for managing the sizes of the files.
> The firewall required too much self configuration to get it right with all the best security practices (ie. What does one do with ICMP.) I was missing something like a template with the defaults that come with UFW, for instance.
I would recommend The Book of PF[0]. While FreeBSD has syntax difference with OpenBSD's pf, this should give you enough insight on how a firewall operates to get a sense of what rules to write.
pm2 has been buggy every time I’ve used it, no matter the OS. Incredibly convenient to begin with but simultaneously unpleasant to use software. Updating environment variables with a deployment has not once ever worked as intended.
Slightly off topic: What's currently the free Linux distribution with the longest support cycle?
For a while I used CentOS 7 on all of those small VMs, because it got security updates for a really long time. With minimal risk of breaking things on updates.
PS: after a bit of research Alma/Rocky Linux are probably the best choices for now. 10 years of support. But are they maintained well?
For a while (a decade+), I was running CentOS on my servers on the same assumption of long time stability and ensuing peace of mind. Then I figured that over such durations, the ecosystem drift becomes significant and keeping applications up to date and running on top of the OS becomes an increasing challenge (with the more "infrastructure" packages like glibc, python/Apache combos, GCC, ... slowly becoming incompatible with the latest applicative stack).
Then I figured that version upgrades were miserable, not just because I had painted myself in a weird corner with ungodly packages mix-ups, but because the upgrade path was always best-effort. I think I gave up during the 6 to 7 transition, as I realised that all I needed was fedora: with yearly or half-yearly updates I have no need to fight the distro's packages: stuff stays current and in working order, major distro upgrades go smoothly, downtime is minimal. I'm not considering going back to any "server distribution" ever.
I see no reason not to go with a rolling release distro for personal servers. Run all the services in containers and have the base OS auto-update itself as often as it needs.
Went with openSUSE MicroOS myself, it updates and reboots almost daily so I can be pretty confident my server is healthy and it's atomic so if something does break and I don't feel like dealing with it, I can just click rollback button from cockpit and deal with it whenever I have time.
Then you also have to auto-update the containers, if it's a public facing service. Either you'll have to build containers yourself or hope the developer pushes a new update whenever the base image has relevant security fixes.
>!!!I see no reason not to go with a rolling release distro!!! for personal servers. Run all the services in containers and !!!have the base OS auto-update itself as often as it needs.!!!
Alma has a few affordances as it's no longer RHEL source compatible, which means it could ship priviledge escalation fixes with new kernel updates faster.
Rocky responded with an extra, optional to enable, security repo to provide mitigations to the exploits while waiting for RHEL to downstream.
Look pretty well maintained to me. If only judging by recent events.
Rocky's docs are also really nice. They aren't as thorough as RedHat's, but they're much more readable and concise, and tend to be written for a less enterprise-y audience.
I don't care much about being fully RHEL compatible, or no ABI changes at all. I just want a system that gets security fixes quickly with as little chances of breaking things as possible.
You are betting that whatever you host doesn't live as long as the upgrade cycle because it'll probably be a pain when the upgrades finally arrive. I'd rather have smaller version jumps more often than a huge jump with everything changing after a long time.
It usually doesn't live until the end of the support cycle. And if it does I will probably migrate it to a fresh VM instead of upgrading the distribution.
Alma and Rocky if you want fully free or have a lot of machines. RHEL if you are okay with registering with them; they give ten machines free access to their updates for each Registered account in their system.
RHEL is definitely the most stable major distribution. Alma and Rocky are essentially downstream clones of RHEL.
I've only been running NixOS (in any serious capacity) for three years, but I have installed it on every computer that I am allowed to install it on now.
It has been the most headache-free Linux I've used, simply because I'm less scared to play with and fix stuff. The fact that rollbacks are trivial and snapshots are automatic, and since everything is declarative in a text file anyway, I am way braver. If I do something like screw up the video driver, or the wifi driver or make it so the system doesn't boot anymore, all I need to do is reboot and choose a previous generation.
Earlier today, I tried to run a simple nix tool a colleague made. 3 hours into the build, it crashed. Something about a missing python import? I ran the exact same ‘nix develop’ again. 2 hours later, it worked.
Keep in mind: this was just a simple rest server. But for some reason it needed to (nondeterministically) build the word from scratch to send that single request.
I have run NixOS for about eight years on server and desktop and been a nixpkgs maintainer. Yes, most of the time I would agree with you. The fact that you get warnings in the terminal for a lot of incompatibilities and changes when upgrading is a really nice touch and upgrades tend to be smooth. I do not use rollbacks much, but when you do need them they are really handy. Having every configuration in a single file makes you more bold to play around with configurations, which felt really empowering when I first got into NixOS, as I knew it could roll things back and I no longer had to keep notes on how each box was set up to refer to in the case of a reinstall or migration.
However, I have had one machine become unbootable as it could no longer mount its encrypted disks after an upgrade, forcing me to mount a rescue image remotely, mount the disks manually, lift the data out, and do a complete reinstall (migrated the box to OpenBSD at that time). Similarly, NixOS once messed up systemd (or vice versa) so badly that I could not even reboot without forcing a power cycle. Lastly, I have had a package break for my use cases by maintainers enabling so many custom flags by default for a package that they enabled one I have never seen enabled by any other packaging team and that then broke RTSP in "funny" ways. Ubuntu did tend to break things like graphics between releases at times back when I used it, but I have never had any other distribution or operating system throw curve balls like the three things I mentioned here.
My general impression of NixOS is that the core is solid, but that nixpkgs just has such a large number of things that it supports that the maintainers struggle to test them all and can not anticipate the interactions between all the packages and options. The default Julia package being so broken that it produced incorrect mathematics due to nixpkgs' insistence on allowing you to swap out the Blas library and also having turned off the unit tests for example springs to mind. This was shipped to end users for a long time before I noticed it by accident by enabling the unit tests and stepped in to clean it up. It all feels very "Gentoo", which was indeed an inspiration for NixOS by the way.
Now, return to that last sentence in the first paragraph that I wrote about feeling empowered to tinker, ultimately, I feel like you should try to resist that urge as it is what pushes you into the untested fractal of possible configurations that NixOS allows you to explore. My other main operating system is OpenBSD, where the mentality is "Stick to the defaults or suffer the consequences"; with NixOS, I feel like everyone's box is more or less a tailored suit, which comes with both its ups and downs.
I run nixOS as well on my home infrastructure (gateway/firewall, a couple of internal servers).
But I have had, uh, non-trivial breakages happen also when I upgrade the system itself to the next yearly release. Non-bootable kernel kind of breakages.
But I will give you that I can just boot from the generation before the upgrade, and it works again. So there's that :)
Eh nix flakes are a nightmare to configure. Far more verbose than a docker compose. They rely on some caches which keep pre-compiled packages and you better make sure you have the caches with the particular flakes you need set up. Yikes
5 years is not a lot. It releases every 2 years, so it requires upgrading at least every 4 years. In the worst case it's just 3 years of support, if you install right before the next release.
ELTS is 10 years and paid. It's great that it exists, but not relevant for my toy projects.
Probably Debian or Ubuntu. The question is...why do you care that much?
I've upgraded Debian stable (both pure and with some cherry-picked backports) and Ubuntu (non-LTS and LTS) systems in place and rarely broken anything, for years and years. When stuff has broken it's been a quick google and then slapping myself for not having read the upgrade guide.
I do generally wait about 2-3 weeks before upgrading, giving time for them to catch stuff that was missed until the great masses were set loose on it.
Not the OP, but I support Ubuntu as desktop and server OS for an engineering collage and have for 10ish years. Some LTS upgrades don't require many changes (mostly minor package name changes) and some take months of work to get rolled out (mostly for workstations, the server upgrades are usually quick.). Not everything gets upgraded every new OS release. If we had to upgrade everything every 6-12 months it would eat up a significant amount of time for our small team.
I've had issues with Ubuntu/Debian upgrades more than once. Some third party binaries breaking with the update. Or some specific config tweaks that break, because the structure of /etc changed too much.
For some small VM with a specific purpose I prefer a distribution that changes as little as possible for as long as possible. Less work, more uptime.
Ive had nothing but issues doing that. I think I’ve had a Debian upgrade actually succeed maybe one time? (After some manual intervention to fix some issue other booting on my work server)
For updates, Debian and Ubuntu are great. For upgrades… not so much for me.
But then you have constant maintenance. I prefer rolling distros, don't get me wrong. But it does mean you will get the latest of every package constantly and some cause problems.
For a box that sits in a corner doing its joband you don't want to pay attention to it's not a good choice IMO. On a desktop you want the latest of everything on and you have time to keep up it's the best.
I need to enable automatic updates, because I don't have the time to manually update. I have a few machines on Open SuSE Tubleweed, and stuff just randomly breaks. A few months ago there was a weird Kernel bug that just froze all of them. They update and reboot every day, and suddenly it all worked well again. A bit too exciting for me :)
> I don’t know why fastfetch always report more memory being used than the actual values. I’ve never seen more than 3GiB used in btop for this server
Probably, it's because of ZFS ARC (Adaptive Replacement Cache). It's similar to Linux's page cache, can be claimed back any moment and different tools name it differently: https://www.linuxatemyram.com/
I'm in the same boat. I have 2 old servers that I let get "too" old, and now I'm afraid to touch them to update them. However, with some of the shenanigans that the Linux distributions are pulling around age verification/attestation, I'm considering bailing on them entirely.
Note, I did try Artix, but when it broke last week after a restart (in which evidently something had gone wrong with an earlier kernel update), and I had to pull out a rescue ISO, I decided I didn't want to mess with that. I switched that machine to Devuan, but the jury is still out for me. I don't have any major complaints, but I'm still in the burn-in phase. :) I'm running Arch on a laptop, but they have been a bit hostile in the community with censorship, so I'm just waiting for a free weekend to blast it and put something else on. I don't want political drama in my software.
This all comes at an interesting time, though. This is the first time that I purchased a new laptop and didn't even let it boot into Windows, but instantly installed Linux. And everything "just worked". And now that I'm excited to try Linux, so many of the big players are embracing the steps to erode privacy (AI everywhere... age attestation/verification... telemetry on by default...). It's sad, and I'm just going to "nope" out of any interactions with them.
FWIW, I once abandoned an Ubuntu server for a decade, and managed to update it painlessly in 20 minutes. That same server is still running today, now with the latest LTS. I think I might have even started with Ubuntu 4, or perhaps 6, and it has been painless all the way through. Perhaps my slow upgrades saved me from early adopter woes :).
I use Debian now much more. With all the supply chain attacks, Debian Stable feels like an absolute jewel, even if there are always a few packages I need to handle separately because I want or need a more updated version. But I love the old school no-nonsense engineering ethos.
I have a machine that's on a Debian installation that I've been steadily upgrading, one Debian stable release at a time, since I originally installed it about a decade ago now. At one point I even copied the entire installation to another disk (just a "dd" from its original SATA SSD to a new NVMe one, plus some partition and filesystem resizing), and I've upgraded the CPU/motherboard/RAM at another time, and it just keeps going reliably. It's fun knowing that the origin of that Debian installation predates every hardware component it's presently running on (with the exception of only the case and power supply).
> something had gone wrong with an earlier kernel update
That's mostly problem of Arch/Artix, they're the bleeding edge, which is not always the best for stability.
But no one said that rolling distro is supposed to always ship latest versions of everything. I've been using Void Linux past months - and while it's a rolling distro, it runs LTS kernel (mainline is also available) and maintainers are more focused on stable versions of apps than on faster updates.
My servers/VMs typically run either FreeBSD or Alpine. A Debian here or there where needed (proxmox, VPS that doesn't support Alpine, corp stuff, etc).
I've also got a couple of test systems running Chimera - going to wait until it hits stable before relying on it too much though. Experimenting a bit with AerynOS too.
I hope FreeBSD has longer supporting cycle. Its release has a supporting life of less than one year, if missing the upgrade window, then later upgrade is more difficult than others such as debian stable.
I've switched to Debian (and since Ubuntu) for my server needs but I remember being obsessed in the mid 2000s with FreeBSD when I was younger. I would spend more time configuring and setting them up than doing anything actually useful on them.
It used to be hard to find dedicated servers or VPSs with any of the BSDs, I think I settled on Panix.com or something?
Before that I remember some company called 15MinuteServers (NAC?) out of NJ I think that offered them. Just kind of rambling down memory lane at this point though.
What is there to be afraid of? Don't you have backups?
Also, debian/ubuntu systems can easily be setup to auto update and reboot on a regular basis, leaving you manual maintenance only for the larger version upgrades.
I love people that aren't afraid to experiment and learn. As someone that hasn't had a formal education in software engineering (just in other kind of engineering) I learned the most by doing and failing.
Formal education doesn't typically emphasize this kind of learning. Univerity CS classes will focus on data structures, algorithms, languages, turing machines and finite automata and how they relate to computability.
If you're a university student and want to learn OpenBSD administration or how to host your own blog, or just how to use VSCode, these are all extracurriculars.
My understanding is that the kernels are mostly equal. I’d be pretty surprised if one had a large impact one way or the other. Any differences I’d chalk up to the userspace program running it.
Can you detail the transition? What were the pain points? I feel like you lose a lot of the selling point of OpenBSD as soon as you start pulling from ports, but how could you do anything productive without it
Ports are sometimes hardened as well, such as Firefox, Chromium, Got (OBSD git alternative, not yet part of base) etc...
But I personally don't really use OpenBSD for security. Sure, good security is important, but for a simple person, I think any updated OS, with good passwords/pubkey auth, good config, being careful etc etc... Is good enough.
OpenBSD is a coherent OS. It's simple (for geeks), and you can use it, by just using the documentation. There's no need for looking up tutorials really, because you don't have to read a 500 page book to understand certain tools, just basic man pages and some computer science knowledge.
With OpenBSD, you go back to a simpler time. Without all the hectic bullshit and an ever-faster pace of constant changes that makes our lives worse, rather than better. The only useful thing it can't do is gaming - with some exceptions, for that I use Windows.
Talking about ports again: OpenBSD comes with batteries included. Not everything though, but you don't really need the ports that much for just a server, if you aren't doing anything complex.
I also use it on desktop/laptop systems, booting it up (yes, it's relatively slow...) always gets me to a state of tranquility. The good ol' days. Maybe that's just my type of brain, but life needs to become simpler again.
Really, what post-2010 information technology has really improved our well-being? Can't think of much.
OpenBSD may have to many rough edges for a desktop system though, even for most geeks. But for those, there is FreeBSD (have it on one laptop). Just get a well-supported machine for that.
All my homelab stuff runs on Proxmox LXC container and fully managed via Ansible non-destructive playbooks.
I just setup Semaphore the other night which adds a web UI to manage Ansible playbooks, it works like this:
1. I host my own Forgejo git repos
2. Semaphore is granted access to the Ansible repo
3. FreshRSS notifies me when a service I am running has new release
4. Check the release note, then run Semaphore to run the ansible-playbook
I could fully automate it all but I have the need to read release notes.
As for the OS, they are Debian 13 Netinst and fully local only, I could run them until the services can no longer run, which the ansible-playbook can spin up another LXC container running Debian 14 or whatever.
The goal is to automate everything as much as possible.
> I don’t know why fastfetch always report more memory being used than the actual values. I’ve never seen more than 3GiB used in btop for this server
My guess would be that fastfetch probably reports actual memory usage while btop probably reports the total usage of all processes. The former is probably higher because of things like filesystem caching
Ha! Mine ran on 18.04 [0] and I migrated it a couple months ago. When I went to take a screenshot for bragging rights, I noticed two other servers 16.10 and 15.04
The applications run just fine, but I don't even know where to start. Apparently I coded them directly into the server, no dev machine!
I was running Ubuntu 16.04; migrated to FreeBSD and I'm all in. Between 16.04 and the current version of Linux; the ecosystem shifted. It's values shifted in ways that did not align with me. This mis-alignment is what motivated me to boot-up FreeBSD. I'm glad I discovered it. I found my happy place again.
It's an incredible journey to take--whether you stick with it or not. Migrating to FreeBSD gives you new eyes into what Linux was, is, and the awesomeness of FreeBSD that is so hard to articulate; like describing the color blue. It must be taken as a whole to appreciate it; and I'm not just saying the OS, it's commands, kernel features, but, the end-to-end compute experience, over time.
If I could draw an equivelent, it would be like when Djistrka savagely destroyed the GOTO statement with a single, short, paper. It took a brilliant mind to articulate that and there has yet to be such a mind to describe the beauty of FreeBSD. So, the best I can do, is just to challenge you to try it.
I've been running FreeBSD at home for a couple of years, and Linux at work (and at home) for 25 years. I'm interested to learn how the Linux ecosystem's values shifted. I figured out pretty early on that Ubuntu wasn't for me (now I usually run Debian, Slackware before that), and I'm wondering if the 'values' issues are Ubuntu specific or if they're some greater problem. I'm not trolling or defending Linux, I'm honestly just curious what you think.
Boggles my mind that people pay money to host hugo static sites on a VPS, which is objectively inferior and harder in every meaningful way compared to hosting for free on GitHub pages or S3+CloudFront.
I did it this way for a long time, but it was mostly a learning/experimenting/fun thing back to have a "proper" server out there that I can run whatever service I wanted on (be it an irc bouncer or whatever). I'm a grownup now and don't have the time/care anymore and just run them out of s3/cloudflare (which was still "fun", but now I don't need to worry about the cve of the day. I don't mind contributing to the centralization of the internet when I'm paying $0/month for pages that nobody visits. Definitely happy to nerd out again if something ever warrants it).
I don’t do it myself, but “objectively inferior in every meaningful way” is a bold claim. It might be harder, but we (geeks) love to do things ourselves.
If someone is willing to use something like Hugo instead of garbage sites like Medium why not use a VPS? For many people working in tech $10/month and free are the same thing.
Personally I get my geek satisfaction from building systems that are rock-solid and require zero maintenance. Not choking on rare opportunities to go viral should they arise is a nice bonus too.
I pay $35/month for a dedicated server for my nothing webpages. One the one hand, I could really host at home for $0/month extra. Or on a VPS for $5/month. I do need my own thing because I run a network testing tool that needs some amount of direct access.
On the other hand, dedicated servers are more fun, even if the cpus I get for $35/month are ancient. Is it $30/month fun? Probably not, but it's near zero given my situation.
At least I'm sensible and don't have an actual colo space to visit.
In the unlikely event I go viral, I'm pretty sure my server can manage serving https at 1gbps, and that's plenty. Maybe TLS is too much though, the cpus are too old for AESNI.
Good on ya. Personally I'm happier with my extra ~$6k, sparing of however many hours of pointless maintenance work over the years, and 100% uptime over past/present/future even if I go into a coma.
some people (myself included) like hosting their own stack for fun or for learning.
There's additional concern with tying your work to something like github it makes it more of a pain to pull it off and put it somewhere else.
I'm not really sure what you mean by objectively inferior. It's trade offs like everything in this field.
As far as harder, I don't really think the lift for a personal VPS is that high. Again it's a fun hobby project for most. It's fun to run your own stack.
If you want to opt into the github cloudflare goodness that's fine they're good services but I wouldn't say it's better or degnegrate others for not doing that.
That's great if that's what you want, but you are commenting in a thread full of people gleefully spouting off about decades-old installations that they self-admittedly have “no idea” how to upgrade. Most people in here would be better off if they admitted to themselves that they are not actually taking advantage of the opportunity to learn, and are instead undertaking a liability.
s3 + cloudfront takes approximately 2 extra steps every deploy, and about 10 extra steps that are easy to screw up at setup time. It's not a trivial drop-in, but yes, once it's done it's _really_ done.
You can make it zero deploy steps beyond git push with CodePipeline, and vibecoding makes the annoying config setup trivial if you know like 20% of what you're doing. There is really zero reason to be using a VPS for this unless you hate money, want your site to choke during once-in-lifetime opportunities to go life-changingly viral, and like contributing to the global population malicious botnets.
The biggest mistake I made was high uptime. arjie.com was up for 10 years plus on a Hetzner VPS so that by the time they wanted to sunset the machine underlying I had no idea what my teenage self had set up. I have the backups but the site hasn’t been up in a decade…
Nowadays I build things so that they move and I have moved things about a bit so I know they work.
"The biggest mistake I made was high uptime"
Quite. I'm old enough to remember machine uptime being a badge of honour.
However, being older and not really wiser, I look for service uptime these days. Yes we did have similar back in the day, that's why MX and the like DNS records exist.
Old school clusters were pretty esoteric but the lessons were learned (split brain n that) and that's why we still argue the toss with kiddies about why a Proxmox cluster with two nodes is fucked and why we recommend an additional "witness".
I don't care that VMware glossed over the whole two node HA cluster thing years ago with a massive bodge. They were wrong then and they are probably still wrong because that nonsense is probably still baked in.
Sorry, slight digression.
High uptime implies no patching. We all love patching.
https://en.wikipedia.org/wiki/Split-brain_(computing)
The more you know!
>a Proxmox cluster with two nodes is fucked and why we recommend an additional "witness".
Reminds me of the three Magi from Evangelion: https://magi.kinta.ma/
1 reply →
My raspberry pi serves only to be the tiebreaker my possible split brain 2 node cluster lol. It is literally called tiebreaker
There is something like live patching.
One reason mainframes and micros are still around us, is that you can change almost everything between hardware and software without downtime.
It is also available in commercial surviving UNIXes, and as paid for feature in some Linux distros, although not to the extent that those grandparent systems are capable of.
19 replies →
In 2012 I took over a Perl project that was running on 25 BSD servers (OpenBSD I think?) that had not been updated / patched since 2000. It was an interesting time.
> Yes we did have similar back in the day, that's why MX and the like DNS records exist.
Care to elaborate? I wanna know more.
1 reply →
two is the right minimum number for a high availability dataplane but three is the right minimum number for a HA control plane.
With that said, if high availability is not a concern then 1 can be just fine.
It's pretty easy to abstract away a proxmox node into a terraform or other type of code based recipe for easy backup / reconstruction / upgrading.
This reminds me of Ise Shrine in Japan, which is completely dismantled then rebuilt every 20 years.
This is top of mind because I recently read Breakneck by Dan Wang. He makes the case that this practice of rebuilding the shrine preserves knowledge that would otherwise have been lost to time. Wang contrasts Ise Shrine with Notre Dame, where rebuilding the roof is apparently quite difficult, perhaps in part due to the loss of knowledge. I'm not familiar enough with either structure to judge whether this is a fair comparison, but I like the principle.
(Edit to add: This is only a minor analogy from the book, which I highly recommend overall.)
Thank you for the recommendation! I love that reference, and particularly because I am fond of the story of the shrine for a different reason https://wiki.roshangeorge.dev/w/Constancy_Preference#Concept...
Indeed, for a VM, high uptime makes little sense, because a reboot takes a few seconds, and an upgrade requires no downtime, just switching the DNS to a new instance.
For a physical machine which you can't easily copy, it's a different story.
We're talking what, 15 minutes to reach post?
I started putting things in a big ansible playbook repo. Don't need to have it fully managed by ansible either I mostly just have setup configured there I still do lots of by hand management.
I have the same. The infra management is in one place, the apps hold their own, and there’s a docs folder on the server where each guy puts his stuff. The install is idempotent deploy scripts. But back then my stuff was more ramshackle.
Sometimes I leave Architectural Decision Records for personal projects. It feels silly but it honestly comes in handy more times than expected
I keep them embedded in the codebase or an artifact right next to the source.
And the key thing is that i dont need too many details at all. A few cues and its all back in my head.
I hear you. On the other hand, not having to mess with something is good. I just make extensive notes in a README somewhere - usually in KeePass right next to the system info.
I stood up a dokuwiki instance recently and then documented how to stand up dokuwiki, haha.
I disabled revision history viewing and have a public portion and a private portion. I use it to track things I'm learning and document rollout procedures and commands I need for things. So far I have rclone backups into S3 Glacier, Tuwunel(Matrix) server deployment with voice/video support, and various little tutorials on server stuff I'm learning.
TLDR use a wiki!
> The biggest mistake I made was high uptime. arjie.com was up for 10 years plus on a Hetzner VPS so that by the time they wanted to sunset the machine underlying I had no idea what my teenage self had set up. I have the backups but the site hasn’t been up in a decade
LLMs have solved this problem, they’ll happily deal with the software archaeology on your behalf. This is the kind of task they really excel at.
You're right, of course. At this point it's inertia. It's been dead a decade.
Personally, I've been running with Caddy in front of Docker (compose) for most of my personal/hobby usage. If it's a straight website, I'll let Caddy serve the contents directly... for "web apps" I'll pretty much containerize all the things and use caddy for TLS termination and reverse-proxy duties to the app running under Docker...
Mostly ~/apps/appname, where each appname has a docker compose file, and the data directories mounted under appname... I can compose down and (s)ftp the data out for hard archives or to move a site/service. I had been running a few VMs under a dedicated server, but switched to separate VPSes on OVH. Only gotcha with OVH is if you want to run mail, you want to avoid the local zone VMs that don't allow mail hosting.
YMMV
I have started using Traefik on one of my projects and it's a nice upgrade from nginx proxy manager. NPM is great, its web gui is noce, but with Traefik all I need to do is write what I want to happen in the docker compose file and that's it.
My home setup is kind of this but it's unraid running the containers and one of them is an nginx thingy specifically designed for reverse proxying other services.
This is more or less how I do it, too. Debating switching from Debian to something like Flatcar which is container-first with an immutable system/OS.
I think FreeBSD has some interesting technical merits, but like it or not Docker is the default for a lot of open source software, and I have neither the time nor the inclination to translate everything to freebsd jails.
I've had my private servers running Arch (managed by Ansible) for the last 5 years but have recently been looking into Talos for the same reasons. Setting up a single node k8s using Talos was actually pretty straight forward. In the end I decided against it just because I couldn't justify the continuous load caused by k8s components when my stuff is only sporadically in use... Instead I now have them on NixOS and I really enjoy the declarative approach (although the language annoys me).
Oh wow. I did the same thing a couple of weeks ago. The server hadn't been updated since ~2015, running a blog on Ghost from that time with node 0.10 installed.
I was a bit rougher though: I just took a backup, then let my Hermes agent (Gemini 3.1 Pro) loose on it. It upgraded everything that needed to be upgraded, patched what needed to be patched, then proceeded to migrate everything to it's most recent equivalents. After that, a fair bit of server hardening was carried out followed by debloating of unused services. Likely would have continued to procrastinate doing this if it wasn't for AI support.
I mean you can update trivially without AI. The problem is the risk of breaking something, which the backup mitigates.
FreeBSD uses ZFS.
I've been using ZFS for about a decade on several systems and can't say enough good things about it: rock solid, feature rich and easy to use are the top benefits.
It really needs more love!
And it's been used by many commercial networking and storage appliances too. Juniper using various devices (routers, firewalls, switches, etc), Citrix Netscreen load balancers, Dell storage Isilon just to name few. Knowing FreeBSD pays dividends if you work enterprise gear. Makes your life so much easier even there is vendor specific UI's and shell, but you know your way around so much better when you encounter any issues which requires using shell and it helps you debugging things you would otherwise not being able to accomplish.
Knowing you way around *BSD how check things, mount a USD drive, collect data & or evidence when need arises is well worth having beyond just surviving bare Linux skills. BSD's are alive and kicking on commercial appliances and devices.
I enjoyed my foray into trying FreeBSD for my personal server. There's something cool, clean, simple and "punk rock" about it. But I gave up as my main pain points were:
- PM2 was buggy on FreeBSD, which I used to manage my processes
- An alternative, using `rc.d` to run daemons was just so hard to get logs working.
- The firewall required too much self configuration to get it right with all the best security practices (ie. What does one do with ICMP.) I was missing something like a template with the defaults that come with UFW, for instance.
> I was missing something like a template with the defaults that come with UFW, for instance.
FreeBSD does include this! It's implemented using IPFW instead of PF. Check out `firewall_type` key in `rc.conf`: https://cgit.freebsd.org/src/tree/libexec/rc/rc.conf?id=8e08...
For a very easy single-machine firewall, one could set `firewall_type=client` or `firewall_type=workstation` if you want to host anything. For the latter, `firewall_myservices` and `firewall_allowservices` control what ports are enabled and who (other networks/IPs) have access to them.
For a very simple NAT gateway, one could set `firewall_type=simple` and then `firewall_simple_(iif|inet|oif|onet)(_ipv6)?` to configure the ISP-side and internal-side interface names and IPv4 and IPv6 network ranges for each.
For more details and to see exactly what each option actually does, check out `/etc/rc.firewall` where this is all implemented: https://cgit.freebsd.org/src/tree/libexec/rc/rc.firewall?id=...
> - PM2 was buggy on FreeBSD, which I used to manage my processes
For supervision?
> - An alternative, using `rc.d` to run daemons was just so hard to get logs working.
The unix way is to use logger(1) If you only want some simple message, or redirect to files using newsyslog(8) for managing the sizes of the files.
> The firewall required too much self configuration to get it right with all the best security practices (ie. What does one do with ICMP.) I was missing something like a template with the defaults that come with UFW, for instance.
I would recommend The Book of PF[0]. While FreeBSD has syntax difference with OpenBSD's pf, this should give you enough insight on how a firewall operates to get a sense of what rules to write.
[0]: https://nostarch.com/book-of-pf-4e
pm2 has been buggy every time I’ve used it, no matter the OS. Incredibly convenient to begin with but simultaneously unpleasant to use software. Updating environment variables with a deployment has not once ever worked as intended.
My main pain points were that it doesn't survive power hits. If your power goes out, it will reboot and ask you to manually fsck the filesystem.
You didn’t use ZFS with FreeBSD?
1 reply →
Slightly off topic: What's currently the free Linux distribution with the longest support cycle?
For a while I used CentOS 7 on all of those small VMs, because it got security updates for a really long time. With minimal risk of breaking things on updates.
PS: after a bit of research Alma/Rocky Linux are probably the best choices for now. 10 years of support. But are they maintained well?
For a while (a decade+), I was running CentOS on my servers on the same assumption of long time stability and ensuing peace of mind. Then I figured that over such durations, the ecosystem drift becomes significant and keeping applications up to date and running on top of the OS becomes an increasing challenge (with the more "infrastructure" packages like glibc, python/Apache combos, GCC, ... slowly becoming incompatible with the latest applicative stack).
Then I figured that version upgrades were miserable, not just because I had painted myself in a weird corner with ungodly packages mix-ups, but because the upgrade path was always best-effort. I think I gave up during the 6 to 7 transition, as I realised that all I needed was fedora: with yearly or half-yearly updates I have no need to fight the distro's packages: stuff stays current and in working order, major distro upgrades go smoothly, downtime is minimal. I'm not considering going back to any "server distribution" ever.
I see no reason not to go with a rolling release distro for personal servers. Run all the services in containers and have the base OS auto-update itself as often as it needs.
Went with openSUSE MicroOS myself, it updates and reboots almost daily so I can be pretty confident my server is healthy and it's atomic so if something does break and I don't feel like dealing with it, I can just click rollback button from cockpit and deal with it whenever I have time.
Everything you listed is the antithesis to managing and maintaining high availability systems
2 replies →
Then you also have to auto-update the containers, if it's a public facing service. Either you'll have to build containers yourself or hope the developer pushes a new update whenever the base image has relevant security fixes.
1 reply →
>!!!I see no reason not to go with a rolling release distro!!! for personal servers. Run all the services in containers and !!!have the base OS auto-update itself as often as it needs.!!!
you do not belong in IT
2 replies →
> But are they maintained well?
Alma has a few affordances as it's no longer RHEL source compatible, which means it could ship priviledge escalation fixes with new kernel updates faster.
Rocky responded with an extra, optional to enable, security repo to provide mitigations to the exploits while waiting for RHEL to downstream.
Look pretty well maintained to me. If only judging by recent events.
Rocky's docs are also really nice. They aren't as thorough as RedHat's, but they're much more readable and concise, and tend to be written for a less enterprise-y audience.
3 replies →
Thanks!
I don't care much about being fully RHEL compatible, or no ABI changes at all. I just want a system that gets security fixes quickly with as little chances of breaking things as possible.
4 replies →
You are betting that whatever you host doesn't live as long as the upgrade cycle because it'll probably be a pain when the upgrades finally arrive. I'd rather have smaller version jumps more often than a huge jump with everything changing after a long time.
It usually doesn't live until the end of the support cycle. And if it does I will probably migrate it to a fresh VM instead of upgrading the distribution.
2 replies →
Alma and Rocky if you want fully free or have a lot of machines. RHEL if you are okay with registering with them; they give ten machines free access to their updates for each Registered account in their system.
RHEL is definitely the most stable major distribution. Alma and Rocky are essentially downstream clones of RHEL.
I would say NixOS, where it is trivial to switch across releases, run software from different releases, and perform rollbacks.
I have been running NixOS on several servers for more than a decade. No reinstalling, upgrading, or any breaks whatsoever.
This is your personal opinion, a rolling release like NicOS is exactly the opposite of an LTS distro.
I actually wonder what would happen to a NixOS installation frozen in time for 5 years that then you want to update to latest all of a sudden.
2 replies →
I've only been running NixOS (in any serious capacity) for three years, but I have installed it on every computer that I am allowed to install it on now.
It has been the most headache-free Linux I've used, simply because I'm less scared to play with and fix stuff. The fact that rollbacks are trivial and snapshots are automatic, and since everything is declarative in a text file anyway, I am way braver. If I do something like screw up the video driver, or the wifi driver or make it so the system doesn't boot anymore, all I need to do is reboot and choose a previous generation.
2 replies →
Earlier today, I tried to run a simple nix tool a colleague made. 3 hours into the build, it crashed. Something about a missing python import? I ran the exact same ‘nix develop’ again. 2 hours later, it worked.
Keep in mind: this was just a simple rest server. But for some reason it needed to (nondeterministically) build the word from scratch to send that single request.
I’ll take a docker system thank you.
I have run NixOS for about eight years on server and desktop and been a nixpkgs maintainer. Yes, most of the time I would agree with you. The fact that you get warnings in the terminal for a lot of incompatibilities and changes when upgrading is a really nice touch and upgrades tend to be smooth. I do not use rollbacks much, but when you do need them they are really handy. Having every configuration in a single file makes you more bold to play around with configurations, which felt really empowering when I first got into NixOS, as I knew it could roll things back and I no longer had to keep notes on how each box was set up to refer to in the case of a reinstall or migration.
However, I have had one machine become unbootable as it could no longer mount its encrypted disks after an upgrade, forcing me to mount a rescue image remotely, mount the disks manually, lift the data out, and do a complete reinstall (migrated the box to OpenBSD at that time). Similarly, NixOS once messed up systemd (or vice versa) so badly that I could not even reboot without forcing a power cycle. Lastly, I have had a package break for my use cases by maintainers enabling so many custom flags by default for a package that they enabled one I have never seen enabled by any other packaging team and that then broke RTSP in "funny" ways. Ubuntu did tend to break things like graphics between releases at times back when I used it, but I have never had any other distribution or operating system throw curve balls like the three things I mentioned here.
My general impression of NixOS is that the core is solid, but that nixpkgs just has such a large number of things that it supports that the maintainers struggle to test them all and can not anticipate the interactions between all the packages and options. The default Julia package being so broken that it produced incorrect mathematics due to nixpkgs' insistence on allowing you to swap out the Blas library and also having turned off the unit tests for example springs to mind. This was shipped to end users for a long time before I noticed it by accident by enabling the unit tests and stepped in to clean it up. It all feels very "Gentoo", which was indeed an inspiration for NixOS by the way.
Now, return to that last sentence in the first paragraph that I wrote about feeling empowered to tinker, ultimately, I feel like you should try to resist that urge as it is what pushes you into the untested fractal of possible configurations that NixOS allows you to explore. My other main operating system is OpenBSD, where the mentality is "Stick to the defaults or suffer the consequences"; with NixOS, I feel like everyone's box is more or less a tailored suit, which comes with both its ups and downs.
I run nixOS as well on my home infrastructure (gateway/firewall, a couple of internal servers).
But I have had, uh, non-trivial breakages happen also when I upgrade the system itself to the next yearly release. Non-bootable kernel kind of breakages.
But I will give you that I can just boot from the generation before the upgrade, and it works again. So there's that :)
Eh nix flakes are a nightmare to configure. Far more verbose than a docker compose. They rely on some caches which keep pre-compiled packages and you better make sure you have the caches with the particular flakes you need set up. Yikes
I don't have data, but my guess would be Debian or Slackware
No
Ubuntu Pro is free for up to 5 systems. 15 years.
Debian LTS/extended LTS
5 years is not a lot. It releases every 2 years, so it requires upgrading at least every 4 years. In the worst case it's just 3 years of support, if you install right before the next release.
ELTS is 10 years and paid. It's great that it exists, but not relevant for my toy projects.
9 replies →
Probably Debian or Ubuntu. The question is...why do you care that much?
I've upgraded Debian stable (both pure and with some cherry-picked backports) and Ubuntu (non-LTS and LTS) systems in place and rarely broken anything, for years and years. When stuff has broken it's been a quick google and then slapping myself for not having read the upgrade guide.
I do generally wait about 2-3 weeks before upgrading, giving time for them to catch stuff that was missed until the great masses were set loose on it.
> The question is...why do you care that much?
Not the OP, but I support Ubuntu as desktop and server OS for an engineering collage and have for 10ish years. Some LTS upgrades don't require many changes (mostly minor package name changes) and some take months of work to get rolled out (mostly for workstations, the server upgrades are usually quick.). Not everything gets upgraded every new OS release. If we had to upgrade everything every 6-12 months it would eat up a significant amount of time for our small team.
3 replies →
> why do you care that much?
I've had issues with Ubuntu/Debian upgrades more than once. Some third party binaries breaking with the update. Or some specific config tweaks that break, because the structure of /etc changed too much.
For some small VM with a specific purpose I prefer a distribution that changes as little as possible for as long as possible. Less work, more uptime.
1 reply →
Ive had nothing but issues doing that. I think I’ve had a Debian upgrade actually succeed maybe one time? (After some manual intervention to fix some issue other booting on my work server)
For updates, Debian and Ubuntu are great. For upgrades… not so much for me.
I had unattended-upgrades cripple our VMs
I mean Ubuntu Pro is free for personal use and it extends the LTS support of 5 years so a total of 10 years afaik.
[dead]
Use a rolling release like Arch and it’s supported forever.
But then you have constant maintenance. I prefer rolling distros, don't get me wrong. But it does mean you will get the latest of every package constantly and some cause problems.
For a box that sits in a corner doing its joband you don't want to pay attention to it's not a good choice IMO. On a desktop you want the latest of everything on and you have time to keep up it's the best.
2 replies →
I need to enable automatic updates, because I don't have the time to manually update. I have a few machines on Open SuSE Tubleweed, and stuff just randomly breaks. A few months ago there was a weird Kernel bug that just froze all of them. They update and reboot every day, and suddenly it all worked well again. A bit too exciting for me :)
1 reply →
> I don’t know why fastfetch always report more memory being used than the actual values. I’ve never seen more than 3GiB used in btop for this server
Probably, it's because of ZFS ARC (Adaptive Replacement Cache). It's similar to Linux's page cache, can be claimed back any moment and different tools name it differently: https://www.linuxatemyram.com/
Ubuntu has no ui vision besides we wanna be:
Sleek like apple
Discoverable like android
Worksuitable like windows
Our usp is:?
I'm in the same boat. I have 2 old servers that I let get "too" old, and now I'm afraid to touch them to update them. However, with some of the shenanigans that the Linux distributions are pulling around age verification/attestation, I'm considering bailing on them entirely.
Note, I did try Artix, but when it broke last week after a restart (in which evidently something had gone wrong with an earlier kernel update), and I had to pull out a rescue ISO, I decided I didn't want to mess with that. I switched that machine to Devuan, but the jury is still out for me. I don't have any major complaints, but I'm still in the burn-in phase. :) I'm running Arch on a laptop, but they have been a bit hostile in the community with censorship, so I'm just waiting for a free weekend to blast it and put something else on. I don't want political drama in my software.
This all comes at an interesting time, though. This is the first time that I purchased a new laptop and didn't even let it boot into Windows, but instantly installed Linux. And everything "just worked". And now that I'm excited to try Linux, so many of the big players are embracing the steps to erode privacy (AI everywhere... age attestation/verification... telemetry on by default...). It's sad, and I'm just going to "nope" out of any interactions with them.
FWIW, I once abandoned an Ubuntu server for a decade, and managed to update it painlessly in 20 minutes. That same server is still running today, now with the latest LTS. I think I might have even started with Ubuntu 4, or perhaps 6, and it has been painless all the way through. Perhaps my slow upgrades saved me from early adopter woes :).
I use Debian now much more. With all the supply chain attacks, Debian Stable feels like an absolute jewel, even if there are always a few packages I need to handle separately because I want or need a more updated version. But I love the old school no-nonsense engineering ethos.
I have a machine that's on a Debian installation that I've been steadily upgrading, one Debian stable release at a time, since I originally installed it about a decade ago now. At one point I even copied the entire installation to another disk (just a "dd" from its original SATA SSD to a new NVMe one, plus some partition and filesystem resizing), and I've upgraded the CPU/motherboard/RAM at another time, and it just keeps going reliably. It's fun knowing that the origin of that Debian installation predates every hardware component it's presently running on (with the exception of only the case and power supply).
> However, with some of the shenanigans that the Linux distributions are pulling around age verification/attestation...
You've been misled.
> something had gone wrong with an earlier kernel update
That's mostly problem of Arch/Artix, they're the bleeding edge, which is not always the best for stability. But no one said that rolling distro is supposed to always ship latest versions of everything. I've been using Void Linux past months - and while it's a rolling distro, it runs LTS kernel (mainline is also available) and maintainers are more focused on stable versions of apps than on faster updates.
My servers/VMs typically run either FreeBSD or Alpine. A Debian here or there where needed (proxmox, VPS that doesn't support Alpine, corp stuff, etc).
I've also got a couple of test systems running Chimera - going to wait until it hits stable before relying on it too much though. Experimenting a bit with AerynOS too.
I hope FreeBSD has longer supporting cycle. Its release has a supporting life of less than one year, if missing the upgrade window, then later upgrade is more difficult than others such as debian stable.
I've switched to Debian (and since Ubuntu) for my server needs but I remember being obsessed in the mid 2000s with FreeBSD when I was younger. I would spend more time configuring and setting them up than doing anything actually useful on them.
It used to be hard to find dedicated servers or VPSs with any of the BSDs, I think I settled on Panix.com or something?
Before that I remember some company called 15MinuteServers (NAC?) out of NJ I think that offered them. Just kind of rambling down memory lane at this point though.
These days it’s fairly straightforward to install on my providers.
I have FreeBSD with Hetzner and OVH. I’ve also used Vultr in the past.
OVH has FreeBSD templates. And most KVM VM/VPS providers will allow console access and mount custom ISO to install whatever you want.
The first time someone explained Docker to me I remember saying, "Oh, you mean a jail?". Not quite, as the article explains. :)
kqueue was a huge win too.
A huge thank you to the FreeBSD developers. I ran my first company for 15 years on FreeBSD with incredible uptime and resilience.
[dead]
I, too, have a server running 16.04 that I'm afraid to update. It currently has an uptime of 1281 days... at this point I'd feel bad rebooting it
dd filesystem to another machine then boot it up with an emulator like qemu and do a trial run
Be careful if you have anything that autostarts that reaches out
Genius.
What is there to be afraid of? Don't you have backups?
Also, debian/ubuntu systems can easily be setup to auto update and reboot on a regular basis, leaving you manual maintenance only for the larger version upgrades.
Backups are not going to help if a reboot breaks something.
1 reply →
You do not reboot systems for regular updates. Only in case of critical kernel updates do you consider it
1 reply →
My oldest server is on 8.04.
> In the end, this is all useless, since most of my traffic comes from AI systems crawling it anyway…
I love people that aren't afraid to experiment and learn. As someone that hasn't had a formal education in software engineering (just in other kind of engineering) I learned the most by doing and failing.
Formal education doesn't typically emphasize this kind of learning. Univerity CS classes will focus on data structures, algorithms, languages, turing machines and finite automata and how they relate to computability.
If you're a university student and want to learn OpenBSD administration or how to host your own blog, or just how to use VSCode, these are all extracurriculars.
The benchmarks are completely off, and a recent version of Ubuntu with sane config would easily beat Freebsd.
My understanding is that the kernels are mostly equal. I’d be pretty surprised if one had a large impact one way or the other. Any differences I’d chalk up to the userspace program running it.
Don’t forget there are serious hardware differences. It’s not apples to apples.
But that’s ok. The author isn’t claiming FreeBSD is way better than Linux. It’s just a comparison of what he had vs what he has now.
Show me those benchmarks
From every benchmark I've seen so far, Linux has always been faster than the BSDs.
For example, look at these benchmarks from 2003[1]. The newest benchmarks I could find[2], [3] point in the same direction.
[1] http://bulk.fefe.de/scalability/
[2] https://matteocroce.it/blog/freebsd_linux_networking/
[3] https://www.phoronix.com/review/freebsd-15-amd-epyc-linux
I recently switched from Debian based servers to OpenBSD and I have never been happier. I wish I would have done it much, much earlier.
Can you detail the transition? What were the pain points? I feel like you lose a lot of the selling point of OpenBSD as soon as you start pulling from ports, but how could you do anything productive without it
Ports are sometimes hardened as well, such as Firefox, Chromium, Got (OBSD git alternative, not yet part of base) etc...
But I personally don't really use OpenBSD for security. Sure, good security is important, but for a simple person, I think any updated OS, with good passwords/pubkey auth, good config, being careful etc etc... Is good enough.
OpenBSD is a coherent OS. It's simple (for geeks), and you can use it, by just using the documentation. There's no need for looking up tutorials really, because you don't have to read a 500 page book to understand certain tools, just basic man pages and some computer science knowledge.
With OpenBSD, you go back to a simpler time. Without all the hectic bullshit and an ever-faster pace of constant changes that makes our lives worse, rather than better. The only useful thing it can't do is gaming - with some exceptions, for that I use Windows.
Talking about ports again: OpenBSD comes with batteries included. Not everything though, but you don't really need the ports that much for just a server, if you aren't doing anything complex.
I also use it on desktop/laptop systems, booting it up (yes, it's relatively slow...) always gets me to a state of tranquility. The good ol' days. Maybe that's just my type of brain, but life needs to become simpler again.
Really, what post-2010 information technology has really improved our well-being? Can't think of much.
OpenBSD may have to many rough edges for a desktop system though, even for most geeks. But for those, there is FreeBSD (have it on one laptop). Just get a well-supported machine for that.
Brillant post ! I love Hetzner's cloud UI. Happy customer for many many years with them.
All my homelab stuff runs on Proxmox LXC container and fully managed via Ansible non-destructive playbooks.
I just setup Semaphore the other night which adds a web UI to manage Ansible playbooks, it works like this:
1. I host my own Forgejo git repos
2. Semaphore is granted access to the Ansible repo
3. FreshRSS notifies me when a service I am running has new release
4. Check the release note, then run Semaphore to run the ansible-playbook
I could fully automate it all but I have the need to read release notes.
As for the OS, they are Debian 13 Netinst and fully local only, I could run them until the services can no longer run, which the ansible-playbook can spin up another LXC container running Debian 14 or whatever.
The goal is to automate everything as much as possible.
> hostname: tauceti
The other Hail Mary reference is on top of HN today.
Well done Andy Weir.
I've had a "Get in loser, we are going to Tau Ceti" on my car's bumper ever since the book came out
> I don’t know why fastfetch always report more memory being used than the actual values. I’ve never seen more than 3GiB used in btop for this server
My guess would be that fastfetch probably reports actual memory usage while btop probably reports the total usage of all processes. The former is probably higher because of things like filesystem caching
I was pretty excited to find out that FreeBSD now supports Podman and OCI containers so now I can move some of my web apps from Linux to FreeBSD. :)
Ha! Mine ran on 18.04 [0] and I migrated it a couple months ago. When I went to take a screenshot for bragging rights, I noticed two other servers 16.10 and 15.04
The applications run just fine, but I don't even know where to start. Apparently I coded them directly into the server, no dev machine!
[0]: https://cdn.idiallo.com/images/assets/daily/98/old_servers.j...
I was running Ubuntu 16.04; migrated to FreeBSD and I'm all in. Between 16.04 and the current version of Linux; the ecosystem shifted. It's values shifted in ways that did not align with me. This mis-alignment is what motivated me to boot-up FreeBSD. I'm glad I discovered it. I found my happy place again.
It's an incredible journey to take--whether you stick with it or not. Migrating to FreeBSD gives you new eyes into what Linux was, is, and the awesomeness of FreeBSD that is so hard to articulate; like describing the color blue. It must be taken as a whole to appreciate it; and I'm not just saying the OS, it's commands, kernel features, but, the end-to-end compute experience, over time.
If I could draw an equivelent, it would be like when Djistrka savagely destroyed the GOTO statement with a single, short, paper. It took a brilliant mind to articulate that and there has yet to be such a mind to describe the beauty of FreeBSD. So, the best I can do, is just to challenge you to try it.
I've been running FreeBSD at home for a couple of years, and Linux at work (and at home) for 25 years. I'm interested to learn how the Linux ecosystem's values shifted. I figured out pretty early on that Ubuntu wasn't for me (now I usually run Debian, Slackware before that), and I'm wondering if the 'values' issues are Ubuntu specific or if they're some greater problem. I'm not trolling or defending Linux, I'm honestly just curious what you think.
[dead]
[dead]
Boggles my mind that people pay money to host hugo static sites on a VPS, which is objectively inferior and harder in every meaningful way compared to hosting for free on GitHub pages or S3+CloudFront.
I did it this way for a long time, but it was mostly a learning/experimenting/fun thing back to have a "proper" server out there that I can run whatever service I wanted on (be it an irc bouncer or whatever). I'm a grownup now and don't have the time/care anymore and just run them out of s3/cloudflare (which was still "fun", but now I don't need to worry about the cve of the day. I don't mind contributing to the centralization of the internet when I'm paying $0/month for pages that nobody visits. Definitely happy to nerd out again if something ever warrants it).
I don’t do it myself, but “objectively inferior in every meaningful way” is a bold claim. It might be harder, but we (geeks) love to do things ourselves.
If someone is willing to use something like Hugo instead of garbage sites like Medium why not use a VPS? For many people working in tech $10/month and free are the same thing.
Personally I get my geek satisfaction from building systems that are rock-solid and require zero maintenance. Not choking on rare opportunities to go viral should they arise is a nice bonus too.
1 reply →
I pay $35/month for a dedicated server for my nothing webpages. One the one hand, I could really host at home for $0/month extra. Or on a VPS for $5/month. I do need my own thing because I run a network testing tool that needs some amount of direct access.
On the other hand, dedicated servers are more fun, even if the cpus I get for $35/month are ancient. Is it $30/month fun? Probably not, but it's near zero given my situation.
At least I'm sensible and don't have an actual colo space to visit.
In the unlikely event I go viral, I'm pretty sure my server can manage serving https at 1gbps, and that's plenty. Maybe TLS is too much though, the cpus are too old for AESNI.
Good on ya. Personally I'm happier with my extra ~$6k, sparing of however many hours of pointless maintenance work over the years, and 100% uptime over past/present/future even if I go into a coma.
3 replies →
some people (myself included) like hosting their own stack for fun or for learning.
There's additional concern with tying your work to something like github it makes it more of a pain to pull it off and put it somewhere else.
I'm not really sure what you mean by objectively inferior. It's trade offs like everything in this field.
As far as harder, I don't really think the lift for a personal VPS is that high. Again it's a fun hobby project for most. It's fun to run your own stack.
If you want to opt into the github cloudflare goodness that's fine they're good services but I wouldn't say it's better or degnegrate others for not doing that.
> for learning.
That's great if that's what you want, but you are commenting in a thread full of people gleefully spouting off about decades-old installations that they self-admittedly have “no idea” how to upgrade. Most people in here would be better off if they admitted to themselves that they are not actually taking advantage of the opportunity to learn, and are instead undertaking a liability.
2 replies →
s3 + cloudfront takes approximately 2 extra steps every deploy, and about 10 extra steps that are easy to screw up at setup time. It's not a trivial drop-in, but yes, once it's done it's _really_ done.
You can make it zero deploy steps beyond git push with CodePipeline, and vibecoding makes the annoying config setup trivial if you know like 20% of what you're doing. There is really zero reason to be using a VPS for this unless you hate money, want your site to choke during once-in-lifetime opportunities to go life-changingly viral, and like contributing to the global population malicious botnets.
30 replies →
Another option is cloudflare pages. Can be coupled with any hub or you can just push html artifacts.