Comment by dedicate

It's still a problem for containers-in-VMs. You can in theory do something with either memory ballooning or (more modern) memory hotplugging, but the dance between the OS and the hypervisor takes a relatively long time to complete, and Linux just doesn't handle it well (eg. it inevitably places unmovable pages into newly reserved memory, meaning it can never be unplugged). We never found a good way to make applications running inside the VM able to transparently allocate memory. You can overprovision memory, and hypervisors won't actually allocate it on the host, and that's the best you can do, but this also has problems since Linux tends to allocate a bunch of fixed data structures proportional to the size of memory it thinks it has available.

That's called memory balooning and is supported by KVM on Linux. Proxmox for example can do that. It does need support on both the host and the guest.

> Is there a VM technology that can make Linux aware that it's running in a VM, and be able to hand back the memory it uses to the host OS?

Isn't this an issue of the hypervisor? The guest OS is just told it has X amount of memory available, whether this memory exists or not (hence why you can overallocate memory for VMs), whether the hypervisor will allocate the entire amount or just what the guest OS is actually using should depend on the hypervisor itself.

The short answer is yes, Linux can be informed to some extent but often you still want a memory balloon driver so that the host can “allocate” memory out of the VM so the host OS can reclaim that memory. It’s not entirely trivial but the tools exist, and it’s usually not too bad on vz these days when properly configured.

It’s one reason i don’t like WSL2. When you compile something which needs 30 GB RAM the only thing you can do is terminate the wsl2 vm to get that ram back.

There’s some problem with networking: if you try to run multiple containers, they won’t see each other. Could probably be solved by running a local VPN or something.

This doesn't look like WSL1. They're not running Linux syscalls to the macOS kernel, but running Linux in a VM, more like the WSL2[0] approach.

[0] https://devblogs.microsoft.com/commandline/announcing-wsl-2/...

In the end they're probably run into the same issues that killed WSL1 for Microsoft— the Linux kernel has enormous surface area, and lots of pretty subtle behaviour, particularly around the stuff that is most critical for containers, like cgroups and user namespaces. There isn't an externally usable test suite that could be used to validate Microsoft's implementation of all these interfaces, because... well, why would there be?

Maintaining a working duplicate of the kernel-userspace interface is a monumental and thankless task, and especially hard to justify when the work has already been done many times over to implement the hardware-kernel interface, and there's literally Hyper-V already built into the OS.

Yeah, it probably would be feasible to dust off the FreeBSD Linux compatibility layer[1] and turn that into native support for Linux apps on Mac.

I think Apple’s main hesitation would be that the Linux userland is all GPL.

[1]: https://docs.freebsd.org/en/books/handbook/linuxemu/

In case you're wondering, the Hypervisor.framework C API is really neat and straightforward:

1. Creating and configuring a virtual machine:

    hv_vm_create(HV_VM_DEFAULT);

2. Allocating guest memory:

    void* memory = mmap(...);
    hv_vm_map(memory, guest_physical_address, size, HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC);

3. Creating virtual CPUs:

    hv_vcpu_create(&vcpu, HV_VCPU_DEFAULT);

4. Setting registers:

    hv_vcpu_write_register(vcpu, HV_X86_RIP, 0x1000); // Set instruction pointer
    hv_vcpu_write_register(vcpu, HV_X86_RSP, 0x8000); // Stack pointer

5. Running guest code:

    hv_vcpu_run(vcpu);

6. Handling VM exits:

    hv_vcpu_exit_reason_t reason;
    hv_vcpu_read_register(vcpu, HV_X86_EXIT_REASON, &reason);

One of the reasons OrbStack is so great is because they implement their own hypervisor: https://orbstack.dev/

Apple’s stack gives you low-level access to ARM virtualization, and from there Apple has high-level convenience frameworks on top. OrbStack implements all of the high-level code themselves.

Using a hypervisor means just running a Linux VM, like WSL2 does on Windows. There is nothing native about it.

Native Linux (and Docker) support would be something like WSL1, where Windows kernel implemented Linux syscalls.

Every container system is "docker-like". Some (like Podman) even have a drop-in replacement for the Docker CLI. Ultimately there are always subtle differences which make swapping between Docker <> Podman <> LXC or whatever else impossible without introducing messy bugs in your workflow, so you need to pick one and stick to it.

Yeah, from a quick glance the options are 1:1 mapped so an

  alias docker='container'

Should work, at least for basic and common operations

OS X pulls some components of FreeBSD into kernel space, but not all (and those are very old at this point). It also uses various BSD bits for userspace.

Good read from horse mouth:

https://developer.apple.com/library/archive/documentation/Da...

„Container“ is sort of synonymous with „OCI-compatible container“ these days, and OCI itself is basically a retcon standard for docker (runtime, images etc.). So from that perspective every „container system“ is necessarily „docker-like“ and that means Linux namespaces and cgroups.

Conceptually similar but different implementations. Containers uses cgroups in Linux and there is also file system and network virtualization as well. It's not impossible but it would require quite a bit of work.

Another really good read about containers, jails and zones.

https://blog.jessfraz.com/post/containers-zones-jails-vms/

BSD jails are architected wholly differently from what something like Docker provides.

Jails are first-class citizens that are baked deep into the system.

A tool like Docker relies using multiple Linux features/tools to assemble/create isolation.

Additionally, iirc, the logic for FreeBSD jails never made it into the Darwin kernel.

Someone correct me please.

Unless you need to have a working video or audio config as well.

Using the Linux filesystem has almost no performance penalty under WSL2 since it is a VM. Docker Desktop automatically mounts the correct filesystem. Crossing the OS boundary for Windows files has some overhead of course but that's not the usecase WSL2 is optimized for.

With WSL2 you get the best of both worlds. A system with perfect driver and application support and a Linux-native environment. Hybrid GPUs, webcams, lap sensors etc. all work without any configuration effort. You get good battery life. You can run Autodesk or Photoshop but at the same time you can run Linux apps with almost no performance loss.

Aren't the syscalls a constant moving target? Didn't even Microsoft fail at keeping up with them in WSL?

FreeBSD and NetBSD do this.

They didn't.

It would probably be slower than just running a VM.

Podman Desktop, and probably other Linux-containers on macOS tools, can already create multiple VMs, each hosting a subset of the containers you run on your Mac.

What seems to be different here, is that a VM per each container is the default, if not only, configuration. And that instead of mapping ports to containers (which was always a mistake in my opinion), it creates an externally routed interface per machine, similar to how it would work if you'd use macvlan as your network driver in Docker.

Both of those defaults should remove some sharp edges from the current Linux-containers on macOS workflows.

Use OrbStack for sometime, made my dev team’s m1 run our kubernetes pods in a much lighter fashion. Love it.

How does it compare to Podman, though?

It’s really awful. There’s a certain size at which you can pivot and keep most of your dignity, but for Docker Inc., it’s just ridiculous.

Still needs a VM. It'll be running more VMs than something like orbstack, which I believe runs just one for the docker implementation. Whether that means better or worse performance we'll find out.

there's still a VM involved to run a Linux container on a Mac. I wouldn't expect any big performance gains here.

Yes, it seems like it's actually a more refined implementation than what currently exists. Call me pleasantly surprised!

My guess is nerdctl and containerd.

Migrate to Orbstack now, and get a lot of sanity back immediately. It’s a drop-in replacement, much faster, and most importantly, gets out of your way.

There's also Rancher Desktop (https://rancherdesktop.io/). Supports moby and containerd; also optionally runs kubernetes.

I have to drop docker desktop at work and move to podman.

I'm the primary author of amalgamation of GitHub's scripts to rule them all with docker compose so my colleagues can just type `script/setup` and `script/server` (and more!) and the underlying scripts handle the rest.

Apple including this natively is nice, but I won't be a able to use this because my scripts have to work on linux and probably WSL

Orbstack

vminitd is the most interesting part of this.

That's my guess too... Colima, but probably doing a VM using the Virtualization framework. I'll be more curious if you can select x86 containers, or if you'll be limited to arm64/aarch64. Not that it really makes that much of a difference anymore, you can get pretty far with Linux Arm containers and VMs.

It would be better for the OrbStack guy if they bought it.

The article says that what was announced is "foundation model frameworks", hence the awkward twist in the title, to get two frameworkses in there.

It’s title case[0].

[0]: https://en.wikipedia.org/wiki/Title_case

Me too - I thought I'd fixed that! Fixed now, thanks.