I really wish they would have expanded on this paragraph in their article further:
> The configuration used here had no jailer. Firecracker's production containment includes dropping privileges and restricting the process through namespaces and seccomp. A deployment using those controls requires a separate assessment of the privileges and resources available after compromising its VMM.
If I'm understanding all of this correctly, a properly jailed and configured firecracker instance would greatly reduce the blast radius of this escape, and I feel as though it would have been good for them to explain what jailing and dropping privileges would do to help. I'd hope most production firecracker deployments would be doing this already.
Not saying that this is any less severe and the root issue is not specific to Firecracker, but to a flaw in the kernel's network code -- they just used an insecure firecracker configuration to demonstrate it. However, I feel like some are taking this as a flaw with firecracker when its not. This is an issue in the kernel networking stack.
That's a good catch. Yes, since they've only proven this up to the jailer boundary, you'd need a second and relatively powerful LPE vulnerability to get real host control out of this.
Thanks to frontier reasoning agents, Linux is completely riddled in ways you can discover 900+ LPE vulnerabilities. [0]
Another wave of vulnerabilities like this are yet to be discovered in Linux. We are just seeing how weak C and C++ based OSes and kernels are when put under hundreds of AI agents.
Nope. The exploit compromises the host kernel. The researchers built an unconventional exploit chain in which they used the host kernel compromise to compromise the host firecracker userspace code, but I see no fundamental reason they couldn't have compromised something else in the host kernel.
Wait, this is also presumably true, come to think of it. I'm a little fuzzy because they document the memory disclosure primitive carefully but not the memory writes they also got from it --- but whatever those are, they're hitting kernel memory? But the exploit they're describing here gets code execution out of Firecracker itself, not the kernel.
yup, the other firecracker escape recently (https://www.coinspect.com/blog/firecracker-escape/) popped the jailer via io_uring (tl;dr firecracker was filtering syscalls appropriately with seccomp, and was filtering the uring _they_ initiated, so attacker just got them to pop a new unfiltered uring, which is rather nice).
Just for what it's worth, that escape is much more situational than PwnEscape (there have been other escapes that depended on nested KVM, as well). Pretty much everyone seems to have been exposed with PwnEscape, because everyone using Firecracker is also using virtio-net.
I am more convinced by the day that the Nova Microhypervisor approach is correct.
Do virtualization with a tiny formally verified microkernel and have complex kernels like Linux run underneath instead of letting a huge, complex, and brittle linux kernel and vm ecosystem control the hypervisor layer.
How is this different from Xen? It's the same approach as Xen or Hyper-V where the hypervisor is its own component, then you boot into a virtualized Dom0 or Root Partition (in Xen and Hyper-V terminology respectively) that can manage the hypervisor underneath it.
This is different from KVM where your hypervisor runs in your Dom0/Root Partition.
This is a much lighter weight (~100k) alternative to Xen focused on baremetal TPM anchored remote attestation, formal verification, and trusted computing applications.
Anyone considering Xen should probably also consider Nova depending on requirements.
The actual main open source repo has had that warning there for 14 years in spite of commercial forks that are deployed in production government and high risk use cases.
This is one of the most useful web tools I've ever seen.
I work on connectors for datasources (mostly DB's. Being able to quickly test things like sorting behavior and how the same function can be emulated across DB's would have made my life so much easier.
> We can PoC the Firecracker VM Escape on that service.
Does this mean you ran the exploit on your service and confirmed it worked? If so, why would you be advertising this, lol. Unless you've already patched?
This is not a new vulnerability, it's just a description of how to exploit the PwnEscape vulnerability in practice on firecracker VMs (disclosed multiple weeks ago and discussed in some less upvoted posts on HN)
The actual vulnerability got very little discussion. It's clever. Because Linux networking people are obsessed with checksum performance, virtio-net plumbs checksum offload metadata through the host/guest driver interface. Guests can thus deliver the host packets along with instructions on what to checksum. The vulnerability is a missing sanity check that allows checksum instructions to instruct the host to overwrite the header length field in the IPv4 header, rather than the checksum field.
Crucially, this happens in the middle of packet processing, so the kernel code is forced to work with two different ideas of how long the IPv4 header is. All hell predictably breaks loose.
I seem to remember there's a system in place to prevent duplicates (where the submission of a duplicate link results in the upvote of the original story), but maybe that's limited to the first 24h? Idk, HN works in mysterious ways
I posted a similar query on the Replit sandbox engine article, but..
How do people feel about the hardware isolation provided by microVMs versus the stronger end of software isolation techniques like gvisor?
Right now, we put a huge amount of stock in "hardware isolation". If that starts falling to agents to any substantial degree, I wonder whether we might end up leaning more on patchable software isolation, similar to the rise of software RAID (though, in userspace this time).
My intuition is that gVisor is never going to be stronger in a fair comparison. Almost all the vulns are in I/O code and every type of container/VM/sandbox has I/O.
Running our microVM's inside of a jailed zone, at least gives me some peace of mind.. It's far from perfect, as a zone escape could be lurking.. but at least it's a layer of defense.
As another commenter noted, the PoC in this article doesn't defeat the Firecracker jailer, which does essentially what a "zone" would do for you. In both cases, that's cold comfort, because even in an unprivileged sandbox environment you still have a huge kernel attack surface to contend with LPEs in.
I would've thought that you could actually have an incredibly strict seccomp policy for the jailer which would require you to find an LPE in something like open(2), which seems unlikely even in the "agent swarm" era.
Security Exploit requires layering with a Sys Admin Permission Exploit. Nuff said.
This so-called Firecracker exploit never explained how the Guest was able to write to the Host OS's Firecracker user mode process. If the write controlled by the guest to the host was that TCP/IP routing stack Exploit in the host OS's Linux kernel, this Firecracker Exploit is meaningless. The Guest already has unlimited control over the host OS.
I (and anyone) can do this exploit in 2-3 minutes using GDB or Visual Studio's C debugger tool. Freeze process, type new pointers into CPU context snap shot, click resume, Exploit done.
You need something more interesting to say than this, because this is an article by the discoverers of this vulnerability, the first general-purpose Firecracker escape that doesn't depend on nested virtualization in recent memory, along with in-depth discussion of exploitability. It's inherently interesting and on-topic and there isn't an alternate source for the same content.
Informative LLM slop is still LLM slop. We need to insist on high standards of content readability as well as substance. Peer pressure and negative reviews are one way we can influence others.
I really wish they would have expanded on this paragraph in their article further:
> The configuration used here had no jailer. Firecracker's production containment includes dropping privileges and restricting the process through namespaces and seccomp. A deployment using those controls requires a separate assessment of the privileges and resources available after compromising its VMM.
If I'm understanding all of this correctly, a properly jailed and configured firecracker instance would greatly reduce the blast radius of this escape, and I feel as though it would have been good for them to explain what jailing and dropping privileges would do to help. I'd hope most production firecracker deployments would be doing this already.
Not saying that this is any less severe and the root issue is not specific to Firecracker, but to a flaw in the kernel's network code -- they just used an insecure firecracker configuration to demonstrate it. However, I feel like some are taking this as a flaw with firecracker when its not. This is an issue in the kernel networking stack.
That's a good catch. Yes, since they've only proven this up to the jailer boundary, you'd need a second and relatively powerful LPE vulnerability to get real host control out of this.
Thanks to frontier reasoning agents, Linux is completely riddled in ways you can discover 900+ LPE vulnerabilities. [0]
Another wave of vulnerabilities like this are yet to be discovered in Linux. We are just seeing how weak C and C++ based OSes and kernels are when put under hundreds of AI agents.
[0] https://openai.com/index/a-thousand-fewer-paths-to-root/
4 replies →
> This is an issue in the kernel networking stack.
Exactly, its not a firecracker issue. Using firecracker without a jailer is like setting your password to "password".
Hang on, what are you getting at?
10 replies →
Nope. The exploit compromises the host kernel. The researchers built an unconventional exploit chain in which they used the host kernel compromise to compromise the host firecracker userspace code, but I see no fundamental reason they couldn't have compromised something else in the host kernel.
Wait, this is also presumably true, come to think of it. I'm a little fuzzy because they document the memory disclosure primitive carefully but not the memory writes they also got from it --- but whatever those are, they're hitting kernel memory? But the exploit they're describing here gets code execution out of Firecracker itself, not the kernel.
6 replies →
yup, the other firecracker escape recently (https://www.coinspect.com/blog/firecracker-escape/) popped the jailer via io_uring (tl;dr firecracker was filtering syscalls appropriately with seccomp, and was filtering the uring _they_ initiated, so attacker just got them to pop a new unfiltered uring, which is rather nice).
Just for what it's worth, that escape is much more situational than PwnEscape (there have been other escapes that depended on nested KVM, as well). Pretty much everyone seems to have been exposed with PwnEscape, because everyone using Firecracker is also using virtio-net.
2 replies →
I am more convinced by the day that the Nova Microhypervisor approach is correct.
Do virtualization with a tiny formally verified microkernel and have complex kernels like Linux run underneath instead of letting a huge, complex, and brittle linux kernel and vm ecosystem control the hypervisor layer.
https://hypervisor.org/
How is this different from Xen? It's the same approach as Xen or Hyper-V where the hypervisor is its own component, then you boot into a virtualized Dom0 or Root Partition (in Xen and Hyper-V terminology respectively) that can manage the hypervisor underneath it.
This is different from KVM where your hypervisor runs in your Dom0/Root Partition.
Xen had a generic guest to host vulnerability disclosed in the last few days: https://codeant.ai/security-research/xenbleed-a-grant-copy-c...
8 replies →
This is a much lighter weight (~100k) alternative to Xen focused on baremetal TPM anchored remote attestation, formal verification, and trusted computing applications.
Anyone considering Xen should probably also consider Nova depending on requirements.
Isn't VMware also that design ?
Xen is formally verified?
1 reply →
I don't know much of anything about Nova but since the late 1990s this has also been the pitch for L4; there are several mature L4's.
Nova is L4 inspired, and interchangable with L4 kernels in Genode.
10 replies →
[dead]
> the Nova Microhypevisor approach is correct
To save people the time:
"This code is experimental and not feature complete. If it breaks, you get to keep both pieces."
The actual main open source repo has had that warning there for 14 years in spite of commercial forks that are deployed in production government and high risk use cases.
See his latest talk which represents a lot of impressive recent work: https://cfp.3mdeb.com/media/bsmconf/submissions/KW3SYT/resou...
I am working on a production OS using it as a foundation.
1 reply →
[flagged]
2 replies →
I have a demo service that entirely relies on Firecracker VMs: ClickBench playground, https://benchmark.clickhouse.com/playground/
It hosts hundreds of database engines - many of them have vulnerabilities, and some of them, like Pandas, are basically open RCEs regardless.
We can PoC the Firecracker VM Escape on that service.
This is one of the most useful web tools I've ever seen.
I work on connectors for datasources (mostly DB's. Being able to quickly test things like sorting behavior and how the same function can be emulated across DB's would have made my life so much easier.
> We can PoC the Firecracker VM Escape on that service.
Does this mean you ran the exploit on your service and confirmed it worked? If so, why would you be advertising this, lol. Unless you've already patched?
This is not a new vulnerability, it's just a description of how to exploit the PwnEscape vulnerability in practice on firecracker VMs (disclosed multiple weeks ago and discussed in some less upvoted posts on HN)
> already discussed at length on HN
Couldn't find it. Care to provide link.
The actual vulnerability got very little discussion. It's clever. Because Linux networking people are obsessed with checksum performance, virtio-net plumbs checksum offload metadata through the host/guest driver interface. Guests can thus deliver the host packets along with instructions on what to checksum. The vulnerability is a missing sanity check that allows checksum instructions to instruct the host to overwrite the header length field in the IPv4 header, rather than the checksum field.
Crucially, this happens in the middle of packet processing, so the kernel code is forced to work with two different ideas of how long the IPv4 header is. All hell predictably breaks loose.
1 reply →
Fixed a month ago upstream
https://github.com/torvalds/linux/commit/6785011f8b16abf40b1...
https://www.pwn.ai/blog/pwnescape
2 replies →
Ah maybe not on HN, I forget where I learned of it. Either way, multiple weeks ago: https://news.ycombinator.com/item?id=50108845
Dupe: https://news.ycombinator.com/item?id=50381397
(Doesn't HN usually deduplicate links? I'm confused)
Duplicates are generally accepted when there was no significant discussion on the original submission.
I seem to remember there's a system in place to prevent duplicates (where the submission of a duplicate link results in the upvote of the original story), but maybe that's limited to the first 24h? Idk, HN works in mysterious ways
This has negative impact on searching older discussions. Instead of 1 result for 1 URL (within few month) getting 2 or even more.
1 reply →
I posted a similar query on the Replit sandbox engine article, but..
How do people feel about the hardware isolation provided by microVMs versus the stronger end of software isolation techniques like gvisor?
Right now, we put a huge amount of stock in "hardware isolation". If that starts falling to agents to any substantial degree, I wonder whether we might end up leaning more on patchable software isolation, similar to the rise of software RAID (though, in userspace this time).
My intuition is that gVisor is never going to be stronger in a fair comparison. Almost all the vulns are in I/O code and every type of container/VM/sandbox has I/O.
FYI, gvisor also uses KVM under the hood; it's not completely orthogonal to [micro]VMs.
That's not the default gvisor option. It can use kvm, but by default (and most common setup in my limited experience) it doesn't.
You mean virtualised hardware ? That's still technically software
I've been building Firehyve, which is a firecracker like microVM solution for illumos and FreeBSD.. https://github.com/TritonDataCenter/rshyve
Running our microVM's inside of a jailed zone, at least gives me some peace of mind.. It's far from perfect, as a zone escape could be lurking.. but at least it's a layer of defense.
As another commenter noted, the PoC in this article doesn't defeat the Firecracker jailer, which does essentially what a "zone" would do for you. In both cases, that's cold comfort, because even in an unprivileged sandbox environment you still have a huge kernel attack surface to contend with LPEs in.
I would've thought that you could actually have an incredibly strict seccomp policy for the jailer which would require you to find an LPE in something like open(2), which seems unlikely even in the "agent swarm" era.
1 reply →
Security Exploit requires layering with a Sys Admin Permission Exploit. Nuff said.
This so-called Firecracker exploit never explained how the Guest was able to write to the Host OS's Firecracker user mode process. If the write controlled by the guest to the host was that TCP/IP routing stack Exploit in the host OS's Linux kernel, this Firecracker Exploit is meaningless. The Guest already has unlimited control over the host OS.
I (and anyone) can do this exploit in 2-3 minutes using GDB or Visual Studio's C debugger tool. Freeze process, type new pointers into CPU context snap shot, click resume, Exploit done.
https://aws.amazon.com/security/security-bulletins/2026-137-...
Firecracker VMs run on AWS. Odd that the article doesn't mention that.
Firecracker is open source and possibly others use it, AWS sells services with it though.
(A more complete way to say this is that Firecracker is the engine built to run AWS Lambda and AWS Fargate).
1 reply →
It has notable adoption in the agent sandbox space where this type of vulnerability is especially volatile.
> The target was Firecracker v1.16.1 on x86-64, with host kernel 7.0.0-1011-aws.
3 replies →
Fly uses it too afaik
2 replies →
[dead]
Not affected https://aws.amazon.com/security/security-bulletins/2026-137-...
The article is light on details, it is LLM slop.
it's really informative. At least someone are break it.
This is pure LLM slop.
You need something more interesting to say than this, because this is an article by the discoverers of this vulnerability, the first general-purpose Firecracker escape that doesn't depend on nested virtualization in recent memory, along with in-depth discussion of exploitability. It's inherently interesting and on-topic and there isn't an alternate source for the same content.
Informative LLM slop is still LLM slop. We need to insist on high standards of content readability as well as substance. Peer pressure and negative reviews are one way we can influence others.
5 replies →