Comment by fpgaminer
2 days ago
If it's a choice between performance and being able to "safely" run sketchy extensions, I'd rather have performance.
2 days ago
If it's a choice between performance and being able to "safely" run sketchy extensions, I'd rather have performance.
A mixture of threads and processes that can be used to match processors, disk I/O, and network interfaces.
A very long time ago, there was once a feature called "Data Blades" which tanked a commercial database vendor. A badly behaving blade could bring down the entire database. Most anyone who has been working on databases for a few decades remembers this and makes a point of either not introducing these sorts of features or making use of processes over threads.
I have not looked at the code referenced in the mentioned project, but thus far I haven't seen a model that could craft a complete SQL parser on its own.
There are a number of problems, and design decisions, that a developer decides on when writing a database that I don't see any current models… just because you have the ingredients does not mean that the stew is edible.
Informix. Michael Stonebraker, to his credit, learned a lot along the way and revised his thinking about database technology and capability after believing that a single engine could be good at everything.
> A very long time ago, there was once a feature called "Data Blades" which tanked a commercial database vendor.
I have no idea what this is and a web search turned up Harbor Freight woodworking tools.
I think it's this: https://www.ibm.com/docs/en/informix-servers/15.0.x?topic=co...
And what you found seems to be "Dado", not "Data".
1 reply →
My first web search (data blades) turned up harbor freight woodworking tools.
Then I made a second web search: data blades database.
That turned up some ibm database software module technology which I assume is what's being discussed.
5 replies →
While PG's behavior doesn't guarantee a lack of data corruption, "an extension crashed, all bets are off, tear everything down" is going to give you a much better fighting chance against data corruption vs the alternative.
In the age of vibe-generated code, I promise you're gonna want the safety.
So load them up in read replicas
Or use webassembly to sandbox them
1 reply →
You have to have a write path. You're gonna want what to be non-sketchy and performant.
What about just not installing sketchy extensions?
I know right? Postgres is not firefox, we don't operate with 25 extensions on all the time. At most we have 1 and normally we have 0.
You could fix probably the sketchy extension issue with WASM.
Limit, not fix.
Do you mean that you don't run sketchy extensions and therefore this doesn't affect you, or that you're ok with data loss due to extension failures?
What about extensions that are not sketchy? Lots of good ones out there.
People are assuming the extensions can't also be rewritten to be good.
Presumably they'd be fine running in a threaded context.
An extension written for a single threaded host system might not work in a multi-threaded context. For example if has global or shared state that isn't protected with locks or similar (which is unfortunately fairly common in c code)
Extensions like pgvector, TimescaleDB would probably need to be ported tho, not sure how much but there are footguns.
1 reply →
Especially since all those sketchy extensions can be rewritten in rust over a weekend and have their bugs fixed as well.
But it isn't.
Threads does not offer any major performance advantage, performance of processes vs threads is virtually the same. The reason the PostgreSQL project is moving towards threads is to make development easier.
> Threads does not offer any major performance advantage
This is very not true. When it comes to parallel queries, a process model adds a ton of overhead. You can't pass pointers between processes because the address space is different. This adds a ton of overhead in a bunch of different places. For example when doing a parallel hash join, Postgres will have each worker build a local hash table. Then it will take all the tuples out of the local hash table and copy them through shared memory to the leader who will then construct a new hash table. This duplicates a lot of work as you have to hash the tuples multiple times.
A lot of getting to Clickhouse level performance was making better use of parallelism.
Passing pointers is not significantly faster than passing offsets into a shared memory pool.
2 replies →
Ok ... you know PostgreSQL supports hash tables in shared memory, right? PostgreSQL could in theory share those if we wanted to. The issue is just that coding anything which uses shared memory is a lot of work.
Additionally the reasons PostgreSQL does not offer Clickhouse performance has very little to do with parallelism. PostgreSQL plans to move to threading but the efforts around imporving OLAP performance are almost entirely unrelated.
14 replies →
MSSQL can handle 32k open connections no need to run a pooler in front of it, can PG do 32k connections and a process for each?
MSSQL shares cached query plans between connections including jitted code, PG cannot do that and the changes needed to make the plans cross process portable would be extensive while sharing between threads is just normal code sharing between threads.
unless you're spawning them for new connections.
Some, but not that much. Switching PostgreSQL to a threaded model will not magically make spawning connections fast. PostgreSQL connections are quite heavyweight.
The reason to use threads is almost entirely about ease of development, not about performance. If you use shrared memory like PostgreSQL does you need to write your own allocators, etc. So much you get for free if you use threads.