Comment by zbentley
4 hours ago
I don’t think that extreme is compatible with the reporting/analytics use case of SQL DBs. Even though entire roles/companies may never touch that kind of SQL, there’s a massive quantity of it out there.
I once worked on a medical records system (with a pretty well designed but necessarily complex schema) where the primary “patient” data object used by most code was fetched by a query that, depending on what associated data you needed, had between 106 and more than 400 relations (across dozens to hundreds of tables) joined together.
And that was CRUDy data-path code. The OLAP/reporting side added zeros to those numbers. Query texts were often hundreds of kilobytes.
Could the same queries be 'planned', compiled down to pipelined KV operations, by the requester? I don't see that this is inherently less capable. You could even use an existing ORM – though I think you can do better when not compiling to something declarative, maybe more like polars.
I feel like databases effectively (/literally) add a JIT, which can mostly figure out what to do, even has accurate heuristics on the distribution of the data, but in exchange you get a less deterministic system, and less intuition for how to query or structure things. It's like, you know when to use a list/map/queue, but you want to focus on the business logic, so just use a smart collections which guess at runtime.
I think you can get this with FoundationDB, I should experiment rather than hypothesizing, but it feels like it would be nicer
It’s theoretically possible to do that kind of planning on the client, but difficult and not worth it compared to letting the database do it. Especially for reporting, there’s another disadvantage: many query planners use runtime statistics from the database to build the plan; synchronizing those onto the client would be difficult and error-prone.
But why bother? If I have a thousand clients that all want to run a query, why compile the plan a thousand times (and build/distribute the local planner to all of the different clients’ platforms) when I could send a query and have the database plan and cache the query once?