Comment by munchbunny
6 hours ago
> Then I offered as part of my design (and from my XP in more than 10yrs of working in products with petabyte datastores) that dealing with so many services connecting to the Data store directly could run into scale issues.
There's a small but substantial number of engineers out there who haven't operated at the kinds of scales where hyperscalers' limits become normal architectural problems and don't have the humility to imagine that it could be the case. (e.g. blob stores do in fact have limits you can hit, and when you operate at petabyte scales you have to anticipate in the architecture that you can hit them for even trivial operations.) I also work on petabyte datastores and have encountered a bunch of those engineers over time.
To be fair though, that's the small minority of engineers I've encountered, and if it wasn't arguing about the types of scale problems unique to petabyte scales, it'd be about some other nuanced subject matter. It's a humility problem.
No comments yet
Contribute on Hacker News ↗