I have had the same experience within the last 18 months. The storage team came back to me and asked me to spread my ultra high throughput write workload across 52 (A-Za-z) prefixes and then they pre-partitioned the bucket for me.
S3 will automatically do this over time now, but I think there are/were edge cases still. I definitely hit one and experienced throttling at peak load until we made the change.
By the way, that happens quite frequently. I regularly ask them about new AWS technologies or recent changes, and most of the time they are not aware. They usually say they will call back later after doing some research.
The whole auto-balancing thing isn't instant. If you have a burst of writes with the same key prefix, you'll get throttled.
Not the OP but I’ve had AWS-staff recommend different prefixes even as recently as last year.
If key prefixes don’t matter much any more, then it’s a very recent change that I’ve missed.
Might just be that the AWS staff wasn't up to date on this
I have had the same experience within the last 18 months. The storage team came back to me and asked me to spread my ultra high throughput write workload across 52 (A-Za-z) prefixes and then they pre-partitioned the bucket for me.
S3 will automatically do this over time now, but I think there are/were edge cases still. I definitely hit one and experienced throttling at peak load until we made the change.
1 reply →
By the way, that happens quite frequently. I regularly ask them about new AWS technologies or recent changes, and most of the time they are not aware. They usually say they will call back later after doing some research.
That’s possible but they did consult with the storage team prior to our consultation.
But I don’t know what conversations did or did not happen behind the scenes.
That's right, same for me as of only a few months ago.