Comment by dagss
3 days ago
I feel like the article is leaving the most important part out ; how to allocate that "offset" (sequence number) which allows consumers to keep tabs on how far they have read and query if there are new messages in the fallback. There are a few schemes, some more creative than others, but it is not a trivial problem to solve without complexity or possibility of lock contention (or races with consumers if you do it the wrong way). Most people I guess have writers acquire a lock, for instance on a single row in a state table, for allocating the next sequence number for an event topic.
What is the best way? Perhaps a tool that reads CDC and writes allocated event sequence numbers to another table would work ok, or would that have long latency?
And that CDC processor should then do the NOTIFY too.
Also on the consumer side in some usecases batching can drastically improve throughput. You don't even use LISTEN/NOTIFY then. Just run the consumer in a loop and each iteration process all new unprocessed messages, and store your last sequence number processed between iterations.
No comments yet
Contribute on Hacker News ↗