Comment by mkhalil
1 day ago
Event-sourcing solves this. And with how cheap storage is, it should be more prevalent in the industry. IMO the biggest thing holding it back is that there isn't a framework that's plug-and-play (say like Next.js is to React) that provides people with that ability.
I've been working on one in Typescript (with eventual re-writes in other langs. like Rust and Go), but it's difficult even coming up with conventions.
Event sourcing is an expensive solution and I don't mean from a storage perspective — it burns engineering cognitive horsepower quickly on things that don't matter. Do it if you're in finance or whatever. Having been burned by my own "let's event source" impulse on data change tracking systems, I now prefer less sophisticated solutions. Figuring out how to deal with slow projections, watching a projection rebuild go from minutes to hours to a few days as a system I expected to handle a few events/minute go to 20 events/second. Fancy caches can't save you if you want to use that vaunted ability to reconstruct from scratch. Event schema evolution also presents difficult tradeoffs: when old events stop having meaning or evolve in meaning you either end up adding on new event subtypes and variants leaving old cruft to accumulate, or you do migrations and edit history on really large tables.
I'd counsel anyone considering event sourcing to use more "low power" solutions like audit logs or soft deletes (if really necessary) first if possible.
Appreciate your perspective, and it makes me wish there was some kind of online 'engineers learning from their mistakes' forum (rare to see "I burned myself"). To hear hard won knowledge distilled like this is a nice reminder to spend ones complexity budget wisely.
Why not keep the projection always updated?
Care to share what you've been working on?