Comment by bushbaba
13 hours ago
Most startups can just scale your traditional separation of compute & storage here though. You’d be shocked how well duckdb against s3 scales for 99.9% of use cases
13 hours ago
Most startups can just scale your traditional separation of compute & storage here though. You’d be shocked how well duckdb against s3 scales for 99.9% of use cases
It's one thing to be able to scale database compute/storage; it's another thing to be able to partition it. It's extremely common for bad queries/access patterns to cause noisy-neighbor effects on other simultaneous accesses to the database, to the extreme of knocking the whole database over with timeouts/OOMs/etc.
Scaling out DB compute can only help with that to a (expensive) point; eventually, you end up wanting to either prevent the bad queries from being added to the system (DBA culture) or ensure that the bad query runs on database infrastructure that doesn't affect other queries. That's why partitioning DB compute (and storage: noisy-neighbor effects from a bad query at the storage layer don't require storage to be running e.g. a BookKeeper or whatever on a server; they can manifest as hot S3 keys or cloud object/block store rate limiting) is a necessary capability if your plan for dealing with a culture of "anyone can add any access pattern they want" is to scale the DB.
Iceberg addresses the hot s3 key prefixes. It places entropy in the prefix path per object so reads/writes are randomized over the bucket key prefix space.
It’s much less of an issue. Also, 99%+ of Aws customers likely don’t have enough QPS to hit the per prefix limits anyway
Iceberg & Delta Lake do partition the data. You can add as many DuckDB servers processing them as you want, each app can have as many as you need. You pay Amazon for all the I/O and let them worry about it scaling it.
I'm not saying there is no point of contention - depending upon your write patterns you have practical limits, but for reading data that is already stored you pretty much just pay more to do more and the rate of that cost is close to linear.
I'm sure there is a scale at which that pattern breaks, and I'm sure 99.9% of people reading this will never operate a system at that scale.
Most software developers now are absolutely ignorant of performance concerns. Just throw more compute at it until it works, and someone else will pay the AWS bill.
And later complain AWS was too expensive after they kept throwing hardware at it rather than writing an efficient query