Comment by tlonny
4 hours ago
I much prefer cursor paginated API requests vs. webhooks. The obvious downside being that in order to not get 429'd you need a respectable poll frequency - meaning you lose reactivity to new events.
Thus I think webhooks still have a place - but as a simple "poke" that can be sent to the client to tell them something has changed - supplementing a default low frequency polling interval.
This gives us the best of both worlds:
1. No need to bother de-duping/retrying pokes - if you miss a webhook you will shortly recover anyway when you next poll. 2. No need for any local-specific tunnelling/tooling - the local app will work just fine with the default poll interval. 3. No need to keep a connection live for each client. 4. All the good stuff OP mentioned in his blog post.
The Gmail API works nicely like this. There's a history.list endpoint where you can see the recent history of message additions and removals, you can query for just the history that's taken place since a specific event's historyId, and you can subscribe to push notifications (that can be delivered by webhook) which just tell you when there are new history events, and you're expected to hit the history.list endpoint to see what's new. Some dropped push notifications aren't a big deal.
Yep, the poke pattern is additionally nice because it means my state reconciliation function is the same when running on cron interval vs event driven.
On the flip side, it helps to have endpoints which have a query param linking to some sort of resource update time stamp. That way you can query to only get those items changed since last poll.
Absolutely. I have been so livid at so many applications for not providing a decent CDC API (and dont forget deletes). Salesforce perhaps is the best out there. Imagine if every application exposed a standard CDC API, the world of integrations would be so much better.
Webhooks are fine, but a pollable API is a must have. The amount of hacks I had to do at work to workaround shitty APIs gives me nightmares.