Comment by m132
3 hours ago
> It was a kernel panic for Tahoe.
Ah, must be something else then.
> normal guests aren't supposed to do that
Oh how I wish Arm didn't let anything like this slip into the architecture spec to begin with! Massive source of pain, especially with protected memory/CCA guests. It's not macOS triggering this in isolation either. Most start up binaries for QNX do this too, somehow also in the GIC init path.
I've looked at how different hypervisors/VMMs handle this and, if this makes that patch set any less hacky, Virtualization.framework, QNX Hypervisor, and (I think) VMware all decode and emulate those instructions in software. Virtualization.framework is a remarkable spaghetti in this regard :)
> Or pre-patch them all to HVC #1 works too. Patching the host Hypervisor.framework sounds quite brittle especially after they moved to a pile of C++
Possibly! IIRC, if HCR_EL2.HCD==1, HVC should trap as undefined instruction. Not sure how much of HCR_EL2 can be set from the user-space, but perhaps this could be the least invasive way.
Simply ignoring the instruction, though, is not enough. I remember in my setup, with HVC handling stubbed out, secondary cores would always fail to start. I suspect this to be the culprit.
The SMP bring-up code would fail to pass pointer authentication on the first indirect branch. It would then immediately pivot into FLEH->SLEH->panic(). panic() shortly would attempt to make an indirect jump itself, hoping to crash the other processors, but instead, getting stuck in a loop of calling itself. This would eventually get caught by a stack overflow guard somewhere in FLEH/SLEH, which would place the core in an infinite loop, and... the system would continue to run with just the boot core. Yo dawg, I heard you like panics :)
> HCR_EL2.HCD
That's not ideal because of:
> Any resulting exception is taken to the Exception level at which the HVC instruction is executed.
instead of trapping to the hypervisor
> I've looked at how different hypervisors/VMMs handle this and, if this makes that patch set any less hacky, Virtualization.framework, QNX Hypervisor, and (I think) VMware all decode and emulate those instructions in software. Virtualization.framework is a remarkable spaghetti in this regard :)
And so does Hyper-V.
> It's not macOS triggering this in isolation either
There are some nightmare cases that SEPOS specifically triggers, such as doing isv=0 accesses to GICR... when using the Apple vGIC handling _that_ becomes truly bizarre.
> Simply ignoring the instruction, though, is not enough
Yeah that's not a great idea
> instead of trapping to the hypervisor
My bad! I mean, ehh, I guess you could maintain a breakpoint in the guest kernel's exception vector table or have QEMU inject its own "zero-level exception handler" whose only purpose would be to capture those HVCs, but that's not as straightforward as I originally thought. And since those PAC calls are expected to set a few Apple-specific registers anyway, using the entitlement or skipping Hypervisor.framework and talking straight to the kernel seem like the only viable options when macOS is the guest.
> There are some nightmare cases that SEPOS specifically triggers, such as doing isv=0 accesses to GICR... when using the Apple vGIC handling _that_ becomes truly bizarre.
Interesting! Are there any resources out there about virtualizing sepOS?
Not much public yet about VRE virtualisation (which includes SEP) at this point.
> whose only purpose would be to capture those HVCs
quite expensive because you get to trap ~ all EL0 -> EL1 priv transitions through the virtualisation infrastructure as the sync handler has a lot going through it
1 reply →