Comment by devnull3
1 month ago
If I have to guess the tuning and optimizations:
SQLITE:
1. Smaller page size (to be specific: max(disk_sector_size, 512))
2. Integer primary keys
3. WAL mode
4. Increase page cache size. This is assuming that most people will click in the top-left 100x100 checkboxes.
Other optimzation:
1. In mem cache of bitmap of 1B bits, which is about 120MB, which is a shared state for all incoming connections.
So it's less smart than that. The smaller page size sounds like a sensible probably a sensible call (that I didn't make).
2, 3 and 4 are spot on.
Theres's two connection pools one for reads one for write. Writes are batched every in a single transaction every 100ms and renders are pushed out to each connected user if there's a change max every 100ms (with back pressure). Reads are read only, and writes are transaction mode immediate.
Here's the schema:
CREATE TABLE IF NOT EXISTS cell(chunk_id INTEGER, cell_id INTEGER, state INTEGER, PRIMARY KEY (chunk_id, cell_id)) WITHOUT ROWID
And the options:
:cache_size 15625 :page_size 4096 :journal_mode "WAL" :synchronous "NORMAL" :temp_store "memory" :foreign_keys false
It's a pretty naive approach (1 billion rows).
Each user render is querying 2000 rows that then get converted to html elements, compressed and sent down on every change (including scroll). But, because renders are at most every 100ms changes effectively get batched.
On the whole this is relying on the brute force of SQLITE rather than anything too clever (eg: Hilbert curves or encoded chunks).
The point is that there could be any number of states. I wanted this to be quite general as for me this is more of a CRUD app demo.
Amazing! This nicely demonstrates capabilities of datastar.
I suggest to put a comment on the page to open dev tools and see the SSE event in action. I think people will appreciate D* even more.
That's a good idea. I should probably mention it's hypermedia too.
At some point I'll write up a blog post. Virtual scroll was surprisingly simple in the end and mostly leverages CSS grid.
The funny part, is it't not really about Datastar (I use a fraction of its features). It does make pushed based hypermedia simple which opens up a lot of options.
Clojure/JVM and SQLITE is where the interesting stuff is happening for me at least. I guess that's the nice thing, Datastar gets you back to programming in your favourite backend language of choice and then gets out of the way.
But it's more than a checkbox, it has color and can be any element, not just checkboxes