← Back to context

Comment by matheusmoreira

5 years ago

> learning the linux syscall stuff

I've been studying this for a while. Turns out Linux has an amazing interface. It's stable and language-agnostic. All you need to do is put the values in specific registers and execute a special instruction. The result comes back in one of those same registers.

The high level documentation is here:

https://man7.org/linux/man-pages/man2/syscall.2.html

https://man7.org/linux/man-pages/man2/syscalls.2.html

https://www.kernel.org/doc/Documentation/ABI/stable/syscalls

https://www.kernel.org/doc/Documentation/ABI/stable/vdso

On Windows there is a similar interface but it is not stable. The system call numbers can change. Developers are supposed to use the good old Microsoft DLLs in order to get anything done. Just like how everyone uses libc on other systems.

Linux is different. The system call binary interface is the Linux interface. So it's actually possible to trash all of GNU and rewrite the entire Linux user space in Rust or Lisp or whatever. It doesn't have to be written in C. It doesn't even have to be POSIX compliant. Could be GUI-focused!

All you need to make any x86_64 Linux system call is this code:

  long
  system_call(long number, long _1, long _2, long _3, long _4, long _5, long _6)
  {
      register long rax __asm__("rax") = number;
      register long rdi __asm__("rdi") = _1;
      register long rsi __asm__("rsi") = _2;
      register long rdx __asm__("rdx") = _3;
      register long r10 __asm__("r10") = _4;
      register long r8  __asm__("r8")  = _5;
      register long r9  __asm__("r9")  = _6;

      /* r8, r9 and r10 may be clobbered but can't be in the clobbers list
         because the compiler won't use clobbered registers as inputs.
         So they're placed in the outputs list instead. */
      __asm__ volatile
      ("syscall"

       : "+r" (rax),
         "+r" (r8), "+r" (r9), "+r" (r10)
       : "r" (rdi), "r" (rsi), "r" (rdx)
       : "rcx", "r11", "cc", "memory");

      return rax;
  }

This is all you need to do anything. You can perform I/O. You can allocate memory. You can obtain your terminal's dimensions. You can perform ioctl's to your laptop's camera. You could make a new programming language today and all it really needs to be complete is this single function. What if instead of having this function the compiler could simply emit code that conform to this binary interface? The language could have a system_call keyword that generates Linux system call code!

Once I realized this I tried to turn it into a library called liblinux... I stopped working on it when I found out the kernel already has an awesome single file header you can include that lets you build freestanding Linux executables for a ton of architectures. They use it on the kernel to build their own tools!

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

It even includes process entry point code! Linux copies the argument and environment vectors to the stack before entering the executable. The process start up code obtains those pointers and passes them to the main function. It also ensures the exit system call is called.

The process entry point is usually called _start because that's what linkers look for by default. In reality the ELF header has a pointer to the program's entry point, the actual symbol doesn't matter. You can tell the linker to set it to any other address or symbol. Also note that it's an entry point, not a function. There is no return address. Allowing that code to terminate results in a segmentation violation. Hence the need to ensure exit is called before that happens.

The only feature that seems to be missing is support for the table of auxiliary values:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

The auxiliary values are placed on the stack immediately after the environment vector. So all you need to do to find this pointer is loop through it until it goes out of bounds. I wrote this code and it works:

https://github.com/matheusmoreira/liblinux/blob/master/start...

  struct auxiliary { Elf64_Off type; Elf64_Off value; };

  static void *after(void *vector)
  {
      void **pointer = (void **) vector;
      while (*pointer++ != 0);
      return pointer;
  }

  int liblinux_start(void *stack_pointer)
  {
      long count;
      char **arguments;
      char **environment;
      struct auxiliary *values;

      count = *((long *) stack_pointer);
      arguments = ((char **) stack_pointer) + 1;
      environment = arguments + count + 1;
      values = after(environment);

      return start(count, arguments, environment, values);
  }

You can just loop over the pointer to the structure until you find one with type equal to AT_NULL. Example here:

https://github.com/matheusmoreira/liblinux/blob/master/examp...

Author here. You would like this project: https://chromium.googlesource.com/linux-syscall-support/ Thank you for reminding me of the joy I felt when I discovered this. I feel like you should publish this and post it on Hacker News. Because too many people who post here hold the viewpoint that SYSCALL is evil and you must link the platform libc dynamic shared object or else you're a very horrible person who deserves to have their binaries broken like Apple did to Go. But they wouldn't feel that way, if they could just see the beauty you described.

  • Thanks! Your projects are so inspiring. I too felt great joy discovering all this. Every time I see someone asking about system calls I respond by writing about everything I know. I usually don't get many replies... So happy to see another person who understands.

    > You would like this project: https://chromium.googlesource.com/linux-syscall-support/

    Yes, I would! I saw references to this library in your source code, specifically your jump slots implementation. I had no idea Chromium had this and I've been meaning to explore it later. I'm gonna do it now.

    > Because too many people who post here hold the viewpoint that SYSCALL is evil and you must link the platform libc dynamic shared object or else you're a very horrible person who deserves to have their binaries broken like Apple did to Go.

    I know what you mean! Using system calls are heavily discouraged by libc maintainers and even users. Using calls like clone will actually screw up the global state maintained by glibc threads implementation. It gets to the point where they don't even offer wrappers for system calls they don't want to support. I don't like it... What's the point of an amazing system call that lets you choose exactly which resources you want to share with a child task if all it's ever used for is some POSIX threads implementation?

    Even the Linux manuals do this for some reason: the documentation I linked in my above post actually describe the glibc stuff as if it was part of the kernel and leaves the actual binary interfaces as an afterthought. Linux manuals also inexplicably host documentation for systemd instead of a generic description of how a Linux init system is supposed to interface with the kernel. It makes no sense to me!

    I even asked Greg Kroah-Hartman about it on Reddit:

    https://old.reddit.com/r/linux/comments/fx5e4v/im_greg_kroah...

    I actually think using the system call interface is better than using the C library. No thread local errno business, no global state anywhere, no buffering unless you do it explicitly, no C standard to keep in mind... It's just so simple it's amazing. It's also stable unlike other operating systems which ship user space libraries as the actual interface. On Linux there's no reason not to use it!

    > I feel like you should publish this and post it on Hacker News.

    I wrote a liblinux library, the README describes part of my journey learning about this system call stuff. Lots of LWN sources!

    https://github.com/matheusmoreira/liblinux/blob/master/READM...

    I've been thinking about expanding on it in order to describe everything I know about the Linux system call interface. You really think I should publish this?

    The reason I didn't post liblinux here is it's in a very incomplete state and actually less practical than the kernel nolibc.h file. I only discovered the header much later into development and figured there was no point anymore since the kernel had a much better solution not only available but in actual use. I ended up rewriting autoconf in pure makefiles instead...

    • We're pretty much on the same page. I'm not sure if I share your enthusiasm for clone(), but I think the canonical Linux interface is what's going to save us from the dark patterns we see in userspace. I think everyone should learn how to use raw system calls. Because the first thought that's going to cross their mind is "wow I thought my C library was doing all these things" and then they're going to want a C library that offers more value than putting a number in the eax register.

      For example if you want to call fork() using asm() then on Linux it's simple:

          int fork(void) {
            int ax;
            asm volatile("syscall" : "=a"(ax) : "0"(57) : "rcx", "r11", "memory", "cc");
            if (ax > -4096) errno = -ax, ax = -1;
            return ax;
          }
      

      But if you want to support XNU, FreeBSD, OpenBSD, FreeBSD, and NetBSD too, it gets a little trickier:

          int fork(void) {
            char cf;
            int ax, dx;
            ax = IsLinux() ? 57 : 2;
            if (IsXnu()) ax |= 0x2000000;
            asm volatile("clc\n\t"
                         "syscall"
                         : "+a"(ax), "=d"(dx), "=@ccc"(cf)
                         : "1"(0)
                         : "rcx", "r11", "memory", "cc");
            if (cf) ax = -ax;
            if (ax > -4096) errno = -ax, ax = -1;
            if (ax != -1) ax &= dx - 1;
            return ax;
          }
      

      Cosmopolitan abstracts stuff like that for you, but right now that's only if you're the kind of person who doesn't need threads. I imagine you are, since folks who do smart things with multiprocessing models like Go and Chromium usually don't want C libraries potentially stepping on their toes. Oh gosh threads. The day I figure out how to do those, will be day the whole world will want to use this thing. But I want people who use Cosmopolitan Libc to know what value it's providing them. I think the best way to do that is by raising awareness of the systems engineering fundamentals like this. Because that's something you're right to point out that the Linux community leadership has room for improvement on.

      2 replies →