Comment by sandeepkd
18 hours ago
It says that the company is co-founded by Postgres creator. I find that bit hard to believe given that there is nothing novel in the article, probably discovery for them. I do understand that everyone has to go through their own journey to learn these things but at the same time when you are running business then seeking professional help isnt a bad idea.
Based on my experience queries like these cannot scale, whatever you do. However if you are already on a path where you had invested a lot in such queries then hire a DBA, if you are not far off then hire an architect to model the data for better performance.
My understanding (possibly flawed) is that Stonebraker isn't directly involved anymore.
The original idea that he worked on with the DBOS people at MIT and Stanford was very different and much, much more ambitious, which is why the name DBOS seems a little out of place now. The original idea was much closer to a "database OS".
Here [1] is the paper, which proposes that "To improve the scalability, security and operability of OSes, we propose a data-centric architecture: designing the OS to explicitly separate data from computation, and centralize all state in the OS into a uniform data model. In particular, we propose using database tables, a simple data model that has been used and optimized for decades, to represent OS state. With the data-centric approach, the process table, scheduler state, flow tables, permissions tables, etc all become database tables in the OS kernel, allowing the system to offer a uniform interface for querying this state."
The team later published another paper based on their prototype work [2].
Instead, they basically implemented Temporal as a client library with Postgres as the state layer. It's good, but only tangentially related to the original vision.
Maybe the long-term plan is an actual database OS, but it kind of looks like they decided they had to pivot to something much simpler, and slapped on an "for AI" like everyone is doing these days.
[1] https://arxiv.org/abs/2007.11112
[2] https://dl.acm.org/doi/10.14778/3485450.3485454
Scale with what? If you have m distinct values in an index, then listing them this way takes m log(n) time, which is fine for many use cases no matter how much data you have.
The way the OP is trying to achieve all the goals by pushing the complexity on the queries/database is what I am referring to as non-scalable as data grows on SQL DB.
> If you have m distinct values in an index, then listing them this way takes m log(n) time, which is fine for many use cases no matter how much data you have.
And NO the runtimes are not right away applicable on machines at scale. You are dealing with DB locks, page sizes, available memory, existing data in memory, queue depth. Experienced folks get paid to short circuit such learnings
Selecting distinct values of a single column with a simple condition is hardly pushing complexity into the database.
Runtimes are usually pretty well applicable at scale, it’s just that most people don’t have a good intuition about asymptomatic notation. Constants and lower order terms matter a lot in practice but are hidden in asymptotic notation.
It's Stonebraker, he has a history of doing this to sell shit to people who don't need it.