← Back to context

Comment by bolangi

5 hours ago

Dreaming of "virtual filesystems everywhere". Hmm, sorta sounds like Plan 9.

Great to have an inside view of wrangling technologies for these behemoth data sets.

Plan 9 had such a powerful model for networked systems using these virtual file systems, it sounds like a fairytale!

Oh, want to use that other machine as a gateway? Just mount its /net.

Oh, want to route audio through another machine? Just mount their soundcard into your /dev.

Oh, your machine is too puny to do the task at hand? Just run “cpu thebigmachine” which transplanted your entire environment over there (all the virtual file systems) so that you can continue doing what you were doing, but using that machine’s CPU and memory.

This solved the problem of having to transplant your setup to the remote machine, which you have with modern SSH. If you wanted a different environment you instead created it locally. Each process har its own virtual file tree with mounts.

There were cool things at the local level too: All the programs would expose virtual file systems to interact with. Text editor? Each window had a directory with files containing window content, current selection, even the UI “tagline” with commands. This meant you could write scripts for your programs in any language, because you just had to interact with files.

A modern take on plan 9 is definitely on my Christmas wishlist!

I wonder if we're doing virtual filesystems wrong.

There is a good reason why traditionally filesystem access was mediated by the OS layer, but there are many use cases where you just want to give processes a different view of what they already can access and it could be done as a library in the same userspace process.

However, for that to work across all the processes in a session we'd need a standard way to install such a hook in all peocesses and that's achievable to some extent using LD preload but falls apart quite rapidly with statically built binaries or different libcs

  • I think there is a good case to be made for these things not to be mediated by the operating system by default.

    In Objective-Smalltalk[1], I can access a file as follows:

       hello ← file:hello-world.txt 
    

    This is structurally the same way I would access a local variable, environment variable, database, remote http server etc.

       hello ← https://example.com/hello-world.txt
       hello ← env:GREETING
       hello ← var:greeting         //   hello defaults to var:hello
    

    etc.

    And you can also introduce shortcuts

       scheme:greeter ← ref:https://example.com/ asScheme.
       hello ← greeter:hello-world.txt
    

    Or

       scheme:greeter ← ref:file:./ asScheme
       hello ← greeter:hello-world.txt
    

    Sending -asScheme to a reference is just a shorthand that actually constructs a composition[2] of a "path relative" store with the underlying store of the original reference. So the following two are identical:

       scheme:greeter ← ref:https://example.com/ asScheme.
       scheme:greeter ← #MPWRelativeScheme{ base: 'https://example/com' } → #MPWURLSchemeResolver{} 
    

    This composition mechanism can be carried further with post-processing, so for example an img-scheme can be constructed by composing an image-decoder store with the previous store

       scheme:img ← #MPWImageDecoderStore{} → scheme:greeter 
       helloPic ← img:wave.png 
    

    And so on and so forth, caching is also a nice example.

    [1] https://objective.st

    [2] https://dl.acm.org/doi/10.1145/3359591.3359729

> virtual filesystems everywhere

Please universe I beg you.

Git is incredibly mediocre. But it's all most people know. It's a version control tool that can't handle binary files; and no GitLFS does not count. The end result is a version control tool that is unable to actually version control all the things you need for a project.

This results in a Meta VCS layer where a ton of critical assets are stored in Docker files and other misery. If you want to re-compile a project for 2015 then good luck and god speed.

Personally I think full toolchains belong in source control. And that you should be able to clone / materialize a repro, yank your network cable, and build. This is how big tech monorepos work. It is TheWay imho.

  • IMO screw that. It's maybe a good way to build software in exactly one environment for exactly one environment, deployment to a corporate server fleet.

    Consider a Linux desktop distro: if every little binary (out of order of magnitude 1000) acted like the center of the universe with gigabytes of build environment and "opinions" galore instead of portability, builds would take much more resources than they already do and parts wouldn't necessarily work together.

  • Visual Studio and Xcode take up tens of gigabytes, are updated often, and include system components. Storing them in VCS is impossible, and would be a waste of disk space.

    • Disagree. They’re stored _somewhwre_ anyway, and they may as well be versioned.

      Putting toolchains in perforce is how it works for lots of C++ shops, the setup instructions are “sync and hit build”, whether there’s a toolchain upgrade required or not

    • You could consider ZFS a VCS, and it can easily store multiple versions (snapshots) of Visual Studio.

      It's not impossible, there just isn't that much demand for it.