Ok so if we take this repo and then the headline from https://news.ycombinator.com/item?id=41085376, we could get an infinitely fast file system by just putting another SQLite filesystem on top.
I think I interviewed with that team (or tried to at least) - it had great concepts, but the limited leadership I met on that one seemed unlikely to actually get to product (I say with 20 years retrospection) - they were stuck on sql when it needed a directory, but the directory couldn't scale to handle filesystems. Both have since been fixed, but I think the idea is well dead in MS
I'm not a big fan of Windows, but I'm a bit sad that the project didn't go anywhere forever and then was cancelled. They had some pretty ambitious and interesting ideas for it.
E.g. the only public beta of Vista that had it enabled used the database as a backing store for Outlook contacts. Each contact was an object in the database. So were photos. And you could link - as I recall, I only read about it back then - people to photos. And then search on that data, of course. Like, "give me all emails, documents, and photos that have to do with person X, from 2022-2023".
But IIRC, the performance was atrocious. They might have been able to fix that, but instead it ended on the junk heap of unrealized ideas.
I remember those Longhorn betas - yes, it was coming apart under feature creep, but that only gave us more to play with and anticipate in the leaks.
A shame that MS is no longer in the mindset of ambition like this. Instead we get adverts and clickbait in the Start Menu, and the worst filesystem search on any OS I've experienced.
Why would you bother with this instead of just making an in memory drive? I remember mounting in memory drives even in MS-DOS back in the early 90s. I did it to move whatever game I wanted to play into the RAM Drive so my load times would be faster.
It turns out if you run SQLiteFS on SQLiteFS, performance increases exponentially, and as the number of SQLiteFS layers increases, seek and read times asymptotically approach zero. In other words, this is the world’s first O(0) file system.
At this rate, with only a few performance tweaks, SQLiteFS should be able to retrieve files before you even ask for them.
The problem is that it sends you files before you need them, which clogs up throughput as you try to figure out what the files are supposed to be for. It's more efficient to slow everything back down to NVMe speeds.
Jokes aside, if I understand correctly it's still assumed the performance increase from the sqlLite is down to the reduced number of OPEN and CLOSE calls. So if that's the case, then it will be as slow as however many sqllites you're running? Meaning it should in theory lose performance?!
For the bored, I was searching for 'database filesystem' and found one by Oracle. I gather it's a way to make editing BLOBs easier. That could make for some interesting recursive things.
It's "a file system interface placed on top of files and directories that are stored in database tables."[0]
Ok so if we take this repo and then the headline from https://news.ycombinator.com/item?id=41085376, we could get an infinitely fast file system by just putting another SQLite filesystem on top.
IMO in this fantasy it should go like (1-.35)^n, not 1-.35n. We must keep our harebrained schemes plausible.
There's a recursive CTE joke in here somewhere
I was just thinking the same thing.. Im supposed to be on holidays but damn it I need to test this.
Reminds me of Microsoft WinFS.
Which was essentially SQL Server as a filesystem.
Project was ultimately cancelled.
https://en.m.wikipedia.org/wiki/WinFS
I think I interviewed with that team (or tried to at least) - it had great concepts, but the limited leadership I met on that one seemed unlikely to actually get to product (I say with 20 years retrospection) - they were stuck on sql when it needed a directory, but the directory couldn't scale to handle filesystems. Both have since been fixed, but I think the idea is well dead in MS
I'm not a big fan of Windows, but I'm a bit sad that the project didn't go anywhere forever and then was cancelled. They had some pretty ambitious and interesting ideas for it.
E.g. the only public beta of Vista that had it enabled used the database as a backing store for Outlook contacts. Each contact was an object in the database. So were photos. And you could link - as I recall, I only read about it back then - people to photos. And then search on that data, of course. Like, "give me all emails, documents, and photos that have to do with person X, from 2022-2023".
But IIRC, the performance was atrocious. They might have been able to fix that, but instead it ended on the junk heap of unrealized ideas.
I remember those Longhorn betas - yes, it was coming apart under feature creep, but that only gave us more to play with and anticipate in the leaks.
A shame that MS is no longer in the mindset of ambition like this. Instead we get adverts and clickbait in the Start Menu, and the worst filesystem search on any OS I've experienced.
2 replies →
Why would you bother with this instead of just making an in memory drive? I remember mounting in memory drives even in MS-DOS back in the early 90s. I did it to move whatever game I wanted to play into the RAM Drive so my load times would be faster.
I use this today on my Raspberry Pis to load the logs and other active IO to memory and avoid wearing out the SD card.
Ok but why not just use a memory drive directly? What is gained by adding SQLite as a layer?
2 replies →
Videogames have gotten so ridiculously huge nowadays, we’ll be at the prosumer/workstation cusp to make this work nowadays.
If SQLite is faster than a normal file system, how fast is SQLite running on “SQLite as a filesystem”?
It turns out if you run SQLiteFS on SQLiteFS, performance increases exponentially, and as the number of SQLiteFS layers increases, seek and read times asymptotically approach zero. In other words, this is the world’s first O(0) file system.
At this rate, with only a few performance tweaks, SQLiteFS should be able to retrieve files before you even ask for them.
The problem is that it sends you files before you need them, which clogs up throughput as you try to figure out what the files are supposed to be for. It's more efficient to slow everything back down to NVMe speeds.
Less humorously, if one were calculating a few files for compilation, or caching for a service, a SqliteFS using :memory: might be just the thing.
Jokes aside, if I understand correctly it's still assumed the performance increase from the sqlLite is down to the reduced number of OPEN and CLOSE calls. So if that's the case, then it will be as slow as however many sqllites you're running? Meaning it should in theory lose performance?!
Obviously adding redundant layers will reduce performance.
Will anyone mention the repo hasn't been updated on 4 years?
When have you last updated your filesystem?
A fair analogy would be: "when was upstream code of the filesystem was last updated?".
To that question, I doubt one would find a polular filesystem that was updated longer than 1 year ago (maybe I'm too liberal).
Looks like last May.
4 replies →
yesterday, zfs.
It works (on Linux)! Should add that I only got to do what it does after following the advice expressed in the error message:
It did not work using `sudo`
For the bored, I was searching for 'database filesystem' and found one by Oracle. I gather it's a way to make editing BLOBs easier. That could make for some interesting recursive things.
It's "a file system interface placed on top of files and directories that are stored in database tables."[0]
[0] https://docs.oracle.com/en/database/oracle/oracle-database/1...
Related: SQLiteDAV - WebDAV server that maps an SQLite database to directories/files.
https://github.com/Airsequel/SQLiteDAV
I wrote this one, I tested with PostgresDB and Oracle , at time:
https://github.com/gbonacini/dbfs
Filesystems are DB like structures these days anyway.
Are they? From my experience from databases and some limited from filesystems I would disagree. Sure, there are some similarities but not that much.
> Filesystems are DB like structures these days anyway.
I think you're confusing the fact that an application uses data structures with that application being like a database.
Other way around. Applications use databases instead of the filesystem.
Where do you get that from? In my practice this was never true and still is not.