Comment by Wowfunhappy
6 years ago
But why are all drivers expected to be "part of the project"? We don't treat userspace Linux software that way. We don't consider Windows drivers part of Windows.
6 years ago
But why are all drivers expected to be "part of the project"? We don't treat userspace Linux software that way. We don't consider Windows drivers part of Windows.
It's pretty simple, once they expose such an API they'd have to support it forever, hindering options for refactoring (that happens all the time). With all the drivers in the tree, they can simply update every driver at the same time to whatever new in-kernel API they're rolling out or removing. And being that the majority of drivers would arguably have to be GPL anyway, and thus open-source, the advantages of keeping all the drivers in tree are high, and the disadvantages low.
With that, they do expose a userspace filesystem driver interface, FUSE. There used to be a FUSE ZFS driver, though I believe it's mostly dead now (But I never used it, so I don't know for sure). While it's not the same as an actual kernel FS driver (performance in particular), it effectively allows what you're asking for by exposing an API you can write a filesystem driver against without it being part of the kernel code.
Given that the kernel is nearly 30 years old, do you not find it slightly incredible that there has been no effort to stabilise the internal ABI while every other major kernel has managed it, including FreeBSD?
There are ways and means to do this. It would be perfectly possible to have a versioned VFS interface and permit filesystems to provide multiple implementations to interoperate with different kernel versions.
I can understand the desire to be unconstrained by legacy technical debt and be able to change code at will. I would find that liberating. However, this is no longer a project run by dedicated amateurs. It made it to the top, and at this point in time, it seems undisciplined and anachronistic.
You know, come to think of it, is there anything stopping Linux from having a... FKSE (Filesystem in Kernel SpacE) standard API?
Presumably, such a thing would just be a set of kernel APIs that would parallel the FUSE APIs, but would exist for (DKMS) kernel modules to use, rather than for userland processes to use. Due to the parallel, it would only be the work of a couple hours to port any existing FUSE server over into being such a kernel module.
And, given how much code could be shared with FUSE support, adding support for this wouldn't even require much of a patch.
Seems like an "obvious win", really.
It's not the context switch that kills you for the most part, but the nature of the API and it's lack of direct access to the buffer cache and VMM layer. Making a stable FKSE leads to the same issues.
That's why Windows moved WSL2 to being a kernel running on hyper-v rather than in kernel. Their IFS (installable filesystem driver) stack screws up where the buffer cache manager is, and it was pretty much impossible to change. At that point, the real apples to apples comparison left NT lacking. Running a full kernel in another VM ended up being faster because of this.
I mean, it doesn't really work that way, you can't just port a userspace program into a kernel module. For starters, there's no libc in the kernel - what do you do when you want to call `malloc`? ;)
With that, I doubt the performance issues are directly because it runs in userspace, they're likely due to the marshaling/transferring from the in-kernel APIs into the FUSE API (And the complexity that comes with talking to userspace for something like a filesystem), as well as the fact that the FUSE program has to call back into the kernel via the syscall interface. Both of those things are not easily fixable - FKSE would still effectively be using the FUSE APIs, and syscalls don't translate directly into callable kernel functions (and definitely not the ones you should be using).
The hard part isn't the "FKSE API", the hard part is for the "FKSE driver" to be able to do anything other than talk to that API. Like, scheduling, talking to storage, the network, whatever is needed to actually implement a useful filesystem.
The problem is that nobody is interested in doing that and that's why we are in this situation in the first place. If Oracle wanted to integrate ZFS into Linux they would just relicense it.
> With that, they do expose a userspace filesystem driver interface, FUSE.
Yes, which Linus has also poo-pooed:
"People who think that userspace filesystems are realistic for anything but toys are just misguided."
I mean, he's right. VFS, VMM, and buffer cache are all three sides of the same coin. Nearly every system that puts the FS in user space has abysmal performance; the one exception I can think of off the top of my head is XOK's native FS which is very very very different than traditional filesystems at every layer in the stack, and has abysmal performance again once two processes are accessing the same files.
1 reply →
And yet people use them all the damn time because they're incredibly useful and even more importantly are relatively easy to put together compared to kernel modules.
Linus is just plain wrong on this one.
1 reply →
The problem with FUSE file systems is not that they aren't part of the kernel's VCS repo, but that it requires a context switch to user-space.
> But why are all drivers expected to be "part of the project"? We don't treat userspace Linux software that way.
It is the policy of linux development at work. Linux kernel doesn't break userspace, you could safely upgrade kernel, your userspace would work nice. But Linux kernel breaks inner APIs easily, and kernel developers take responsibility for all the code. So if a patch in memory management subsystem broke some drivers, kernel developers would find breakages and fix them.
> We don't consider Windows drivers part of Windows.
Yeah, because Windows kernel less frequently breaks backward compatibility in kernel space, and because hardware vendors are ready to maintain drivers for Windows.
You can license both kernel modules or FUSE implementation any way you see fit. That's a non-issue.
https://www.kernel.org/doc/html/latest/process/license-rules...
It seems that some people are oblivious to the actual problem, which is some people want their code to be mixed into the source code of a software project without having to comply with the rightholder's wishes, as if their will shouldn't be respected.
> We don't consider Windows drivers part of Windows.
I'm not sure you can commit your source code to the windows kernel project.
No, no one wants to force ZFS into the Linux kernel. I think anyone agrees that it needs to be out-of tree the way thinks are currently.
The problem is the nature of changes, and people questioning if there is any good _technical_ reason why some of the changes need to be the way they are done.
Because running proprietary binaries in kernel space is not a good idea nor is it compatible with the vision of Linux?
ZFS isn't proprietary it's merely incompatible with the gpl.