Comment by Guid_NewGuid
3 years ago
I think a strong test a lot of "let's use Google scale architecture for our MVP" advocates fail is: can your architecture support a performant paginated list with dynamic sort, filter and search where eventual consistency isn't acceptable?
Pretty much every CRUD app needs this at some point and if every join needs a network call your app is going to suck to use and suck to develop.
I’ve found the following resource invaluable for designing and creating “cloud native” APIs where I can tackle that kind of thing from the very start without a huge amount of hassle https://google.aip.dev/general
The patterns section covers all of this and more
This is a great resource but the RFC-style documentation says what you SHOULD and MUST do, not HOW to do it ...
I don't believe you. Eventual consistency is how the real world works, what possible use case is there where it wouldn't be acceptable? Even if you somehow made the display widget part of the database, you can't make the reader's eyeballs ACID-compliant.
Yeah, I can attest that even banks are really using best effort eventual consistency. However, I think it is very difficult to reason about with systems that try to use eventual consistency as an abstraction. It's a lot easier to think about explicitly when you have one data source/event that propagates outwards through systems with stronger individual guarantees than eventual consistency.
IMO having event streams as first class is the best way to think about things. Then you don't need particularly strong guarantees downstream - think something like Kafka where the only guarantee is that events for the same key will always be processed in order, and it turns out that that's enough to build a system with clear, reliable behaviour that you can reason about quite easily.
> if every join needs a network call your app is going to suck to use and suck to develop.
And yet developers do this every single day without any issue.
It is bad practice to have your authentication database be the same as your app database. Or you have data coming from SaaS products, third party APIs or a cloud service. Or even simply another service in your stack. And with complex schemas often it's far easier to do that join in your application layer.
All of these require a network call and join.
> It is bad practice to have your authentication database be the same as your app database.
No, this is resume-driven-development, Google-scale-wannabe FUD. Understand your requirements. Multiple databases is non-trivial overhead. The only reason to add multiple databases is if you need scale that can't be handled via simple caching.
Of course it's hard to anticipate what level of scale you'll have later, but I can tell you this: for every tiny startup that successfully anticipated their scaling requirements and built a brilliant microservices architecture that proactively paved the way to their success, there's a 100 burnt out husks of companies that never found product market fit because the engineering team was too busy fantasizing about "web-scale" and padding their resume by overengineering every tiny and unused feature they built.
If you want to get a job at FAANG and suckle at the teat of megacorporations who's trajectory was all based on work done in the early 2000s, by all means study up on "best practices" to recite at your system design interview. On the other hand, if you want to build the next great startup, you need to lose the big co mentality and start thinking critically from first principles about power to weight ratio and YAGNI.
> And yet developers do this every single day without any issue.
And users suffer through unresponsive interfaces and long load times every single day...
Most of our cloud hosted request/responses are within the realm of 1-10ms, and that's with the actual request being processed on the other side. Unless there's a poorly performing O(N) stinker in the works, most requests can be served with most latency being recorded user->datacenter, not machine to machine overhead. This article is a lot bonkers.
> Pretty much every CRUD app needs this at some point and if every join needs a network call your app is going to suck to use and suck to develop.
_at some point_ is the key word here.
Most startups (and businesses) can likely get away with this well into Series A or Series B territory.
thanks a lot for this comment. I will borrow this as an interview question :)