Comment by chisleu
5 days ago
> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots.
As of when? According to internal support, this is still required as of 1.5 years ago.
5 days ago
> You don’t have to randomize the first part of your object keys to ensure they get spread around and avoid hotspots.
As of when? According to internal support, this is still required as of 1.5 years ago.
I think there is some nuance needed here. If you ask support to partition your bucket then they will be a bit annoying if you ask for specific partition points and the first part of the prefix is not randomised. They tried to push me to refactor the bucket first to randomise the beginning of the prefix, but eventually they did it.
The auto partitioning is different. It can isolate hot prefixes on its own and can intelligently pick the partition points. Problem is the process is slow and you can be throttled for more than a day before it kicks in.
> but eventually they did it
They can do this with manual partitioning indeed. I've done it before, but it's not ideal because the auto partitioner will scale beyond almost anything AWS will give you with manual partitioning unless you have 24/7 workloads.
> you can be throttled for more than a day before it kicks in
I expect that this would depends on your use case. If you are dropping content you need to scale out to tons of readers, that is absolutely the case. If you are dropping tons of content with well distributed reads, then the auto partitioner is The Way.
He's not talking about the prefix, just the beginning of the object key.
The prefix is not separate from the object key. It's part of it. There's no randomization that needs to be done on either anymore.
and indeed the bucket is not separate from the object key. the API separates it logically "for humans" but it's all one big string