Comment by hexagonal-sun

3 months ago

Hello!

For the past 8 months, or so, I've been working on a project to create a Linux-compatible kernel in nothing but Rust and assembly. I finally feel as though I have enough written that I'd like to share it with the community!

I'm currently targeting the ARM64 arch, as that's what I know best. It runs on qemu as well as various dev boards that I've got lying around (pi4, jetson nano, AMD Kria, imx8, etc). It has enough implemented to run most BusyBox commands on the console.

Major things that are missing at the moment: decent FS driver (only fat32 RO at the moment), and no networking support.

More info is on the github readme.

https://github.com/hexagonal-sun/moss

Comments & contributions welcome!

Cool project, congrats. I like the idea with libkernel which makes debugging easier before going to "hardware". It's like the advantages of a microkernel achievable in a monolithic kernel, without the huge size of LKL, UML or rump kernels. Isn't Rust async/awat depending on runtime and OS features? Using it in the kernel sounds like an complex bootstrap challenge.

  • Rust's async-await is executor-agnostic and runs entirely in userspace. It is just syntax-sugar for Futures as state machines, where "await points" are your states.

    An executor (I think this is what you meant by runtime) is nothing special and doesn't need to be tied to OS features at all. You can poll and run futures in a single thread. It's just something that holds and runs futures to completion.

    Not very different from an OS scheduler, except it is cooperative instead of preemptive. It's a drop in the ocean of kernel complexities.

    • Yeah, for example embassy-rs is an RTOS that uses rust async on tiny microcontrollers. You can hook task execution up to a main loop and interrupts pretty easily. (And RTIC is another, more radically simple version which also uses async but just runs everything in interrupt handlers and uses the interrupt priority and nesting capability of most micros to do the scheduling)

      3 replies →

    • I find it interesting that this fulfills some of Dennis Ritchie's goals for what became the STREAMS framework for byte-oriented I/O:

      > I decided, with regret, that each processing module could not act as an independent process with its own call record. The numbers seemed against it: on large systems it is necessary to allow for as many as 1000 queues, and I saw no good way to run this many processes without consuming inordinate amounts of storage. As a result, stream server procedures are not allowed to block awaiting data, but instead must return after saving necessary status information explicitly. The contortions required in the code are seldom serious in practice, but the beauty of the scheme would increase if servers could be written as a simple read-write loop in the true coroutine style.

      The power of that framework was exactly that it didn't need independent processes. It avoided considerable overhead that way. The cost was that you had to write coroutines by hand, and at a certain point that becomes very difficult to code. With a language that facilitates stackless coroutines, you can get much of the strengths of an architecture like STREAMS while not having to write contorted code.

    • Ok, I see. I spent a lot of time with .Net VMs, where you cannot simply separate await from the heavy machinery that runs it. I now understand that in a kernel context, you don't need a complex runtime like Tokio. But you still need a way to wake the executor up when hardware does something (like a disk interrupt); but this indeed is not a runtime dependency.

      EDIT: just found this source which explains in detail how it works: https://os.phil-opp.com/async-await/

    • There’s got to be some complexity within the executor implementation though I imagine as I believe you have to suspend and resume execution of the calling thread which can be non-trivial.

      5 replies →

  • This has been a real help! The ability to easily verify the behavior of certain pieces of code (especially mem management code) must have saved me hours of debugging.

    Regarding the async code, sibling posts have addressed this. However, if you want to get a taste of how this is implemented in Moss look at src/sched/waker.rs, src/sched/mod.rs, src/sched/uspc_ret.rs. These files cover the majority of the executor implementation.

> no networking support

Would something like Smoltcp be of help here? https://github.com/smoltcp-rs/smoltcp

Great project either way!

How do you decide which sys calls to work on? Is is based on what the user space binaries demand?

  • Yip, I panic whenever I encounter a syscall that I can't handle and that prompts me to implement it.

    Yeah, I was thinking of integrating that at some point. They've done a really nice job of keeping it no_std-friendly.

    • Original author of smoltcp here! I couldn't have imagined how much of a cornerstone of the Rust ecosystem it would become. Very excited to see it get used like this, and yeah, #![no_std] use cases were the original and primary motivation to build the stack in the first place.

Love the MIT license. If this were further along we could use this as the foundation of our business without having to "give back" device drivers and other things.

  • This should be the sort of red flag to take note of. There’s an LLVM fork for every esoteric architecture now and this sort of thinking will lead to never being able to run your own software on your own hardware again. A reversion to the dark ages of computing.

  • MIT licensed code is a gift. A gift indeed doesn't require the recipient to give back anything related to the gift.

    A "gift" requiring GPL-like conditions isn't really a gift in the common sense. It's more like a contractual agreement with something provided and specific, non-negotiable obligations. They're giving while also asserting control over others' lives, hoping for a specific outcome. That's not just a gift.

    People doing MIT license are often generous enough where the code is a gift to everyone. They don't try to control their lives or societal outcomes with extra obligations. They're just giving. So, I'm grateful to them for both OSS and business adaptations of their gifts.

    • While the FSF's vision for the GPL is clear, the GPL itself is not so powerful that it is more than a "gift" that has some terms if you want to do certain things you are not obligated to do. It is like a grant that enforces some reasonable conditions so the money isn't just misappropriated. I wouldn't give that to a friend for their birthday, but I think it's reasonable that powerful organizations should not be free to do whatever they want. Not that the GPL is perfect for that use, but it's good.

    • > It's more like a contractual agreement with something provided and specific, non-negotiable obligations.

      The obligation is not to the author of the code, it is to the public. MIT-style licenses are gifts to people and companies who produce code and software, copyleft licenses are gifts to the public.

      I don't give a shit about the happiness of programmers any more than the happiness of garbage collectors, sorry. I don't care more that you have access to the library you want to use at your job writing software for phones than I care that somebody has access to the code on their own phone. You're free to care about what you want, but the pretense at moral superiority is incoherent.

      It is non-negotiable. GPL is basically proprietary software. It's owned by the public, and all of the work that you do using it belongs to the public. If you steal it, you should be sued into the ground.

      7 replies →

    • MIT is throwing a free party where food and drinks are paid for, and copyleft is where food is paid for but you BYOB. Both are fine, so what's the problem?

      3 replies →

    • I agree, and even if a company doesn't give back, they further the popularity and sustainability of the project. Isn't Python an MIT-like license (PSFL)? As well as React and Godot? And Tensorflow is also permissive with Apache 2.0, corrrect?

      1 reply →

    • A gift where the recipient can remove the freedoms that they've been enjoying themselves is a bad deal for ensuring those freedoms are available to everyone. A permissive license is a terrible idea for a F/LOSS kernel.

      This is the paradox of tolerance, essentially.

      Also, seeing F/LOSS as a "gift" is an awful way of looking at it.

      4 replies →

    • Thay "holier than thou" attitude from BSD/MIT proponents seems like some kind of ego talking. Freedom to deny others the freedom you had is not noble.

      GPL is a gift that keeps giving.

      2 replies →

  • Do you think soup kitchens and food banks should only serve food to those who volunteer? MIT is a perfectly fine FOSS license.

    • No, but if someone takes the free food and builds a business by selling it to others, without giving anything back to the original places, it harms everyone other than the person doing that.

      F/LOSS is not a charity or a gift, so your analogy is not appropriate. It is a social movement and philosophy with the goal of sharing knowledge and building software for the benefit of everyone. It invites collaboration, and fosters a community of like-minded people. Trust is an implicit requirement for this to succeed, and individuals and corporations who abuse it by taking the work of others and not giving anything back are harmful to these goals. Copyleft licenses exist precisely to prevent this from happening.

      MIT is a fine license for many projects, but not for an operating system kernel.

      4 replies →

  • I take this as an oblique critique of TFA's choice of license. What's it to you? Why must we all use the GPL always in order to satisfy busybodies?

    • >> I take this as an oblique critique of TFA's choice of license. What's it to you? Why must we all use the GPL always in order to satisfy busybodies?

      Thank you for reading it correctly. I originally had a </sarcasm> to make sure nobody thought I liked the license choice. What's it to me? Well someone posted it to HN here so we could comment on it, so I did.

      I think the MIT license has its place, but IMHO it does not belong on an OS like that. Reason is indicated in my original comment.

      1 reply →

Very impressive and I like how accessible the codebase is. Plus safe Rust makes it very hard to shoot yourself on the foot, which is good for outside contributions. Great work!

After you got the busybox shell running, how long did it take to add vim support? What challenges did you face? Did you cross-compile it?

  • BUt it's not "safe" because it's mixed with assembly

    • This has been discussed ad nauseam and this adds nothing new. There's value in the memory safety for the majority of the code even if there are some escape valves ("unsafe" keyword, assembly).

    • There are no programs written in 100% safe Rust—the std library is written with unsafe as needed. But typically the majority of lines in the program—sometimes even all outside std or some well-audited foundational creates—are safe. Those lines can not directly cause any unsoundness, which has tremendous value.

Congratulations on the progress. If I may ask, I'm curious what considerations have motivated your choice of licence (especially since pushover licences seem extremely popular with all kinds of different Rust projects, as opposed to copyleft).

  • I’ve pretty much only seen MIT and to a lesser extent GPL on most open source projects. Would you expect a different license?

  • Copyleft doesn't work well with Rust's ecosystem of many small crates and heavy reliance on libraries alongside static linking.

    If one library be GPLv2 and the other GPLv3 they couldn't be used together in one project. LGPL solves nothing because it's all statically linked anyway. And yes, one could licence under both under the user's choice but then GPLv4 comes out and the process repeats itself, and yes one could use GPLv2+ but people aren't exactly willing to licence under a licence that doesn't yet exist and put blind faith into whoever writes it.

    Using anything but a permissive licence is a good way to ensure no one will lose your library and someone will just re-implement it under a permissive licence.

    C is a completely different landscape. Libraries are larger and the wheel is re-invented more often and most of all dynamic linking is used a lot so the LGPL solves a lot.

    • Correct me if I'm wrong, but I think all of these are solved problems.

      LGPL should only pose a problem if you explicitly want your program to be used with non-free software. And then only if said non-free software doesn't give you a way to rebuild it yourself, should you want to modify the LGPL program. (So not a problem for open-core or public-source projects, either.)

      If, for some reason, you insist on allowing static linking for all the projects, I think the MPL allows this. (People often seem to think of the LGPL, but it's not the only weak-copyleft licence around.)

      Otherwise, when releasing your project under GPLv3+, you don't have to put blind faith into the FSF; you can designate a proxy which will decide whether the new version should be allowed for your project or not. This proxy can be yourself, or it can be a different organisation you choose to trust. Plus, I'm pretty sure the GPL allows you to make linking exceptions of your liking.

      2 replies →

  • What is a "pushover" license?

    • 'Pushover licence' is a licence which may grant freedom, but doesn't care to protect it. One may modify software under a pushover licence and release their modifications as non-free software. Another, more common name is 'permissive licence'.

Impressive work! Do you have any goals, other than learning and having fun?

Also how does it's design compare with Redox and Asterinas?

How does android compatibility look? Can this be compiled to WebAssembly and run in browser?