Comment by jtwaleson

1 year ago

I have a first attempt at a sync engine for my app, but it's very primitive. Just a websocket that sends updates based on database triggers. If you miss one, you have to do a full reload. I know I'll need something better in a year or so.

Any advice on what route to take with creating a sync engine for a product like mine? Self-hosted, single binary web app (Rust) + Postgres db. Frontend is based on VueJS. I've looked at the readme of Yjs and was considering that. I'm a solo dev for now.

I'm tempted to feed Cursor this description of the reverse engineered solution of Linear, but I doubt it'll be successful.

5 comments

jtwaleson

Reply
satvikpendem  1 year ago

I'm using Loro as the CRDT as well as Iroh for byte transfer, works well. You can look at ElectricSQL as a Postgres sync engine but it won't do conflict resolution for you and it's hard doing CRDT operations on relational databases on general.

Look into these as well:

https://www.typeonce.dev/article/how-to-implement-a-sync-eng...

https://www.sandromaglione.com/newsletter/lessons-from-imple...

Same author, not necessarily sure why it's on two different domains with different content but they open sourced their sync engine. If you're interested in this topic, I'd follow. Their newsletter as they have great stuff.

jasonjmcghee  1 year ago

Yjs isn't a sync engine, it's a data structure for managing distributed concurrent updates and ensuring they are conflict free.

Whether you use it feels orthogonal to the problem you're describing.

---

For a minimal scope solution, have you considered making a table in your database where you log each update? Then you can keep an id of your most recent update locally and on websocket reconnection ask for the updates after your current change.

Similar to how in-app notifications work.

---

For local-first, you can use things like:

https://tinybase.org/ https://electric-sql.com/ https://livestore.dev/

But they are pretty foundational. You use them as your storage layer in the front end. So worth considering the scope of the change.

  • jtwaleson  1 year ago

    Thanks, that helps! Like I said I had only very briefly looked at Yjs.

    The thought of an "updates" table has crossed my mind yes, but after some time you want a "materialized view" instead of replaying the history from the start, and that's where it gets complicated ;)

    I'll take a look at those alternatives. I'd rather have something stable than having to re-invent the wheel. Thanks again!