Comment by cmrdporcupine

17 days ago

"Because the system is, well, a hash table with a lock in front of it."

This actually surprises me? I was a very early employee/contractor working on this product but left after 3 months. But this wasn't the shape it had when I was there. It would have been better described as a CoW tree of various sorts, which lets reads and writes advance on separate transactions and reconciles at commit with what I recall was fairly minimal lock-holding time.

When I was there there was no global lock being held while a WASM "reducers" was executing. They operated on their own separate transaction in their own thread.

I haven't looked at their source since -- and I really don't want to because I work on similar things I don't want IP contamination as last I looked they had some sort of weird license -- but there's been .. 3 years? of development on it since and I know Tyler well enough and he hired smart-ish people.

I had a lot of criticisms of what he was making and the way it was being made... but ehh....I would want to confirm in source that this claim is accurate, it makes me a bit suspicious of the accuracy and fairness of the rest of TFA.

5 comments

cmrdporcupine

Reply
cmrdporcupine  17 days ago

Ok, so I had a clanker go look. It actually sounds as bad as the article says. For some reason they apparently lock things while "reducer"s are running.

Robot brain summarized:

"Your historical description may be completely accurate. Your request that the author substantiate the claim was warranted. But the source substantiates it extremely clearly. The blog’s concurrency conclusion is right. Its “hash table,” fairness, “cannot sync WAL,” and “questionable to call it a database” language is deliberately contemptuous and sometimes technically sloppy."

touche

  • cloutiertyler  16 days ago

    I think you mean it sounds as good as the article says it is. SpacetimeDB is single threaded by design, just like TigerBeetle by the way. The reason we changed from MVCC to single threaded design is that it's simply faster for almost all workloads. Measurably so! With MVCC we were doing more bookkeeping than actual work and perpetually causing cache coherency issues.

    "Technically sloppy" is an understatement. The article just gets several things plan incorrect about the durability guarantees, etc.