Comment by avhception
6 days ago
I run FreeBSD in my homelab, too! One reason is the stellar ZFS support, but the simple fun of doing stuff differently is definitely a thing, too. And I like FreeBSD jails.
For me, the balance between all the overhead of the "cattle, not pets" approach and the manual way is the a README.md file for basic setup, and then having Ansible stand up the rest of the configuration. The host is configured as a Jail host, then individual services live inside the jails. Creating and configuring the jails is also done through Ansible. Overall, I really like the setup. I can individually SSH into each jail to allow easy debugging, I can snapshot the jails, and data lives on a special ZFS subvolume that I mount into each jail at "/bucket". This way, I can throw away the jail at any time, fire up Ansible, and have everything up and running again in no time.
> And I like FreeBSD jails.
If you don't know about them already, you may be interested in service jails (forthcoming[1] in 15):
> A service jail shares the complete filesystem tree directly with the host (the jail root path is /) and as such can access and modify any file on the host, and shares the same user accounts with the host. By default it has no access to the network or other resources which are restricted in jails, but they can be configured to re-use the network of the host and to remove some of the jail-restrictions.
* https://docs.freebsd.org/en/books/handbook/jails/#service-ja...
* https://docs.freebsd.org/en/books/handbook/jails/#service-ja...
* https://man.freebsd.org/cgi/man.cgi?query=rc.conf&manpath=Fr...
[1] https://www.freebsd.org/releases/15.0R/schedule/
Sounds interesting, but it sounds like that would mean installing the service software and it's dependencies into the root filesystem. I'm relatively sure I don't want that, as it would create a big mess on the host. I have stuff like Nextcloud in my jails, and wouldn't want to install PHP and all of it's deps outside the dedicated filesystem of the jail.
But it's very cool to see continued development, jails are such an awesome feature!
If I didn't need CUDA support, I'd be on FreeBSD all the way. No systemd, built-in "containers" before they were cool, basically just good 'ole UNIX.
You can pass a CUDA device through to a bhyve vm.
In practice, it's not as easy as it may sound.
I've done something like this in the past, it works really well. Have you used Poudriere? I never tried it, but it sounds promising. Ansible is a good idea as well. I just wrote some shell scripts that parsed a file with some packages and hooks to set up the jails.
These days I have my FreeBSD server providing NFS for a k3s instance on a different box.
Yes, I have used Poudriere. Every time I want to talk about it, I have to look up how exactly that is spelled! My usecase was that I needed stuff to be linked against newer PostgreSQL client libs than were in quarterly.
I thought about using Linux and simply providing storage from FreeBSD using NFS, but that would require 2 machines or a VM that connects to the host using NFS or something... meh.
Sorry, I don’t know why I googled around and thought “Poudriere” was the one.
I was thinking of Bastille, which is for managing jails
1 reply →