Comment by adityaathalye
7 hours ago
Just got started again on my "Poor Man's Bitemporal Data System" (discussed here: https://news.ycombinator.com/item?id=45118585 ).
Holidays nuked all the hot-cached context in my head. I spent a few days just spinning wheels until it repopulated. But the basic idea works now!
Much testing and benchmarking work remains to make sure it's not going to lose data, and that it won't denial-of-service itself (because object-map -> facts fan-out is big).
Also a second giant blog post is due (following the one discussed above). Lots of notes have accumulated.
It will be fun even if the concept ultimately crashes and burns to the ground :)
In which case, there's always datomic and xtdb :D
(def repl-facts
[{:alt.site.evalapply/meta
{:description "Root namespace for a named site."}}
{:alt.site.evalapply/features
{:paid #{:alt.site-feature/feat-1
:alt.site-feature/feat-2
:alt.site-feature/feat-3}
:trial #{:alt.site-feature/feat-4}
:complimentary #{}
:available #{}}}
{:alt.site.evalapply/users
{:authorised #{:alt.user.evalapply/user-1
:alt.user.evalapply/user-2
:alt.user.evalapply/user-3}
:unauthorized #{:alt.root.*}}
:alt.user.evalapply/user-1 {:name {:first "Wiley"
:last "Coyote"}
:roles #{:alt.role.evalapply/owner}}
:alt.role.evalapply/owner {:rw #{:alt.user.evalapply/*}
:ro #{}}}])
(assert-facts! (#'user/system-state)
::app
:alt.root/user-1
(uuid/v7)
repl-facts)
(redact-facts! (#'user/system-state)
::app
:alt.root/user-1
(uuid/v7)
(subvec repl-facts 2))
(read-now! (#'user/system-state)
::app)
(edit: add note about upcoming blog)
Love the idea! Read only the beginning of the first "giant blog post" so maybe it's discussed there, but in Sqlite doesn't WAL mode give you a kind of super simple, super basic temporal data system? If you somehow kept all WAL entries 'as is', it should be possible to reconstruct the state of the db at any point in the past?
Thanks!
> but in Sqlite doesn't WAL mode give you a kind of super simple, super basic temporal data system
Yes-ish.
For one, it is unitemporal. For another, it is for SQLites own transactions, not of the individual datums. Litestream is the way to replicate WAL into an object store / elsewhere. litestream-vfs is also looking good!
https://litestream.io/ , https://litestream.io/reference/vfs/ (announced here: https://fly.io/blog/litestream-vfs/ )
I'm trying to emulate the data assert/redact properties of append-only bitemporal-by-design data stores like XTDB. The giant blog post builds up the intuition from scratch. Or at least tries to.
So my system is going to be: bitemporal data, enforced by SQLite schema and application checks, along with WAL replication for point-in-time transaction backup/restore. Both are entirely distinct, and unaware of each other.