Comment by vhiremath4
4 days ago
I once was the CTO of a company that serviced about 100k requests per day across all our services. We grew to millions and eventually 10's of millions, but, somewhere along the way, an engineer on our team decided he wanted to build a queue off LISTEN/NOTIFY semantics in order to take advantage of strong consistency with the rest of our data model, which seemed reasonable given LISTEN/NOTIFY is not that hard to understand and we did need consistency guarantees for this workflow and this would remove the need for yet another place data got stored and transfered.
In practice, this eventually ended up being very awkward because extending the functionality (since we had "built" it) and had to work around internal pg semantics (we should have just moved off much sooner). It also did not scale well. We ended up getting a ton of disk contention on our RDS instance in non-obvious ways, and the vacuum runs on that table was a nightmare. Additionally, it was hard to get other engineers to really debug and take ownership of the system because they automatically viewed a queue (very easy to understand) implemented in a foreign way (off pg internals) as something "scary". It was emotional, not rational, but we are emotional beings, and I do not blame them. These were good engineers with a lot of other things on their plates.
Obviously, this is all hand-wavy without discussing the internal schema, indeces, etc. that we had set up, but my main takeaway with core technology from this experience was to always reach for the dumb, expected, simple thing. Even if it adds another moving piece in the infra stack. Unless I need very strong data consistency guarantees, it's always better to use something like SQS, Redis queues, etc. where the understanding is that it is just a queue (or at least the API contract suggests simplicity), and then everything needs to work around it.
The fewer mechanistic responsibilities per core data store, the better in my experience.
Not saying this in a mean way:
It seems to me this can be boiled down to "using things without understanding how they work doesn't scale". Yes, vanilla listen/notify doesn't scale. But the OP actually figured out how to make it scale. So your engineers don't have to.
As the community builds, over time, distributed systems, it also understands which brick can do what, and it turns out that starting with less bricks and adding some when you actually need them makes for healthier systems.
Nah this isn't mean at all. I think this is the correct takeaway. I actually felt like I understood how the system worked and it wasn't that difficult for me, but I also understood how, as we were adding engineers who were all very under water, the idea of learning our queueing system to modify a core feature it powered seemed like a big mental context switch (bigger than it actually was). I noted this in another comment, but I am likely discounting how big of an effect our scale/growth impacted peoples' ability to adapt this system. That said, I will pretty much always choose the simple/dumb/easy to understand thing from the start even if it adds another moving component.
Also the results given it the article are still worse than redis with zero work and no risk of breaking later, both in nb of req and latency.
And you can move redis to another server next later if needed to split the db and notification load.
1 reply →
It is so rare that a new moving part beats other factors. But the one thing I will say is that there's tremendous advantages to of all things, your queuing system being independent of other architectural pieces : it's the one component built to natively store and forward, so if you can keep its lifecycle separate then you can harness that to decouple other systems from each other during updates, troubleshooting, patch windows etc.
Great point.
> It is so rare that a new moving part beats other factors
Fair! I will restate that we were growing very fast, so that's probably an outlier environmental factor that I am potentially overly discounting. If that's not the case (likely for most startups), then maybe this is less of a problem.
I feel this is a management problem, and I disagree with the takeaway about adding a piece to the infra stack. "My devs don't want to work on part of our stack" is not a valid reason to add a new network node imo. Just make them work on it.
What is cheaper? Forcing my team to work on something they don't want to or adding a network node (that happens to also be the industry standard)? I used to think in absolutes about these kinds of things, but I came to realize along the way that people management is a process of learning which fires to take on. If I can avoid stripping away someone's autonomy, I will always do that even if I know they are not doing the correct thing (unless the wrong thing is in the critical path of the company).
The problem still needs to be solved (queue working/being up/being built around), but I am ok with the solution that might not be the best engineering solution but still delivers on the product requirements and extends the team's happiness (these systems need to be owned). These were not lazy or incompetent engineers.
Pretty sure if you didn't run the DB on networked disks you would have been fine.