Comment by cmrdporcupine
3 years ago
Concurrent state management is hard, and I remain disappointed that software transactional memory -- and pushing a DB-style approach to non-DB workloads generally -- hasn't caught on.
I think most "application" type programs would benefit from being able to manage state in terms of higher level transactional operations and let their runtime take care of serializing and avoiding deadlock and race conditions. Developers in those spaces have come to rely on garbage collection in their runtimes, I see no reason why they couldn't come to also rely on a fully isolated MVCC transaction model (and only get access to non-transactional memory in exceptional circumstances). Bonus points if said transactions tie back to the persistent RDBMS transaction as well.
There are too many minefields in manual lock management and the tools remain fairly low level. Ownership management in languages like Rust helps, following a discipline like an actor/CSP approach helps, etc. but in the end if there's shared state and there's parallel work, there's potential for trouble.
You might be interested in Joe Duffy’s retrospective on Microsoft’s failed experiment to integrate STM into .NET:
https://joeduffyblog.com/2010/01/03/a-brief-retrospective-on...
You really only need one paragraph from that:
> What do we do with atomic blocks that do not simply consist of pure memory reads and writes? (In other words, the majority of blocks of code written today.)
If you could just get programmers to stop mutating, you could get STM (which incidentally would give you back the ability to mutate)
Recently learned about Verse: https://dev.epicgames.com/documentation/en-us/uefn/verse-lan...
It features "speculative execution" which practically is STM. Known names like Tim Sweeney and Simon Peyton Jones are behind that.
Thanks, skimmed a bit of the first half but will get into it more after my work day.
So, going through this a bit more on lunch break; I think the root of some of the disillusionment faced here -- and probably the lack of success in STM at this level generally -- is that they tried to do too much.
I don't necessarily want the whole memory model of the runtime or VM to offer transactions necessarily. What I think is a good idea is to offer an overall framework -- within which higher level applications can be written -- that brings transactional semantics with it. Basically a set of collections and data transformation and communications and process coordination libraries that work in harmony with an underlying MVCC storage layer -- rather than baking a transactional atomic keyword down to the syntactical level of the language or the memory model of the VM/runtime.
Put another way: I wouldn't necessarily give users the STM facilities. I would use lower level STM facilities to construct a higher level toolkit, and only expose that. Basically an RDBMS in-process -- without the SQL language boundary -- to be frank.
Yes, users could escape it easily, and start doing inconsistent things. But that's on them, same as any other framework.
Provide the pattern and tools in a nice coherent box and don't try to take over the whole world.
1 reply →
It's a good day when I get to use STM.
While the primary benefit is in thinking "these things should happen together or not at all" rather than thinking about locks, there's another feature I always forget about, which is retrying.
Retrying let's you 'nope' out of a transaction, and try again as soon as something changes without busy-waiting.