Comment by imglorp
1 year ago
Harder than you'd think, given a couple of requirements, but there are off the shelf products like AWS's QLDB (and self hosted alternatives). They: Merkle hash every entry with its predecessors; normalize entries so they can be consistently hashed and searched; store everything in an append-only log; then keep a searchable index on the log. So you can do bit-accurate audits going back to the first ledger entry if you want. No crypto, just common sense.
Oddly enough, I worked at a well known fintech where I advocated for this product. We were already all-in on AWS so another service was no biggie. The entrenched opinion was "just keep using Postgres" and that audits and immutability were not requirements. In fact, editing ledger entries (!?!?!?) to fix mistakes was desirable.
> The entrenched opinion was "just keep using Postgres" and that audits and immutability were not requirements.
If you're just using PG as a convenient abstraction for a write-only event log, I'm not completely opposed; you'd want some strong controls in place around ensuring the tables involved are indeed 'insert only' and have strong auditing around both any changes to that state as well as any attempts to change other state.
> In fact, editing ledger entries (!?!?!?) to fix mistakes was desirable.
But it -must- be write-only. If you really did have a bug fuck-up somewhere, you need a compensating event in the log to handle the fuck-up, and it better have some sort of explanation to go with it.
If it's a serialization issue, team better be figuring out how they failed to follow whatever schema evolution pattern you've done and have full coverage on. But if that got to PROD without being caught on something like a write-only ledger, you probably have bigger issues with your testing process.
Footnote to QLDB: AWS has deprecated QLDB[1]. They actually recommend using Postgres with pgAudit and a bunch of complexity around it[2]. I'm not sure how I feel about such a misunderstanding of one's own offerings of this level.
[1] https://docs.aws.amazon.com/qldb/latest/developerguide/what-...
[2] https://aws.amazon.com/blogs/database/replace-amazon-qldb-wi...
Yeah. I'm surprised it didn't get enough uptake to succeed, especially among the regulated/auditable crowds, considering all the purpose built tech put into it.
I think you're forgetting how many businesses are powered by Excel spreadsheets. This solution seems too advanced and too auditable.
I'll just leave that here for no particular reason at all:
https://www.sec.gov/enforcement-litigation/whistleblower-pro...
Better hurry, Elon is gonna dismantle the SEC in about 45 days
Importantly, the SEC is empowered to give 10-30% of the money siezed via whistleblowing too the whistle blower.
> Merkle hash every entry with its predecessors; normalize entries so they can be consistently hashed and searched; store everything in an append-only log;
Isn’t this how crypto coins work under the hood? There’s no actual encryption in crypto, just secure hashing.