Comment by tripplyons
2 days ago
For those who aren't aware, OpenAI has a very similar batch mode (50% discount if you wait up to 24 hours): https://platform.openai.com/docs/api-reference/batch
It's nice to see competition in this space. AI is getting cheaper and cheaper!
DeepSeek has gone a bit different route - they give automatic 75% discount between UTC 16:30-00:30
https://api-docs.deepseek.com/quick_start/pricing
Yes, this seems to be a common capability - Anthropic and Mistral have something very similar as do resellers like AWS Bedrock.
I guess it lets them better utilise their hardware in quiet times throughout the day. It's interesting they all picked 50% discount.
Inference throughout scales really well with larger batch sizes (at the cost of latency) due to rising arithmetic intensity and the fact that it's almost always memory BW limited.
50% is my personal threshold for a discount going from not worth it to worth it.
Bedrock has a batch mode but only for claude 3.5 which is like one year old, which isn't very useful.
One open secret is that batch mode generations often take much less than 24 hours. I've done a lot of generations where I get my results within 5ish minutes.
It can depend a lot on the shape of your batch to my understanding. A small batch job can be tasked out a lot quicker than a large batch job waiting for just the right moment where capacity fits.
The latest price increases beg to differ
Only because Flash was mispriced to start with. It was set too cheap compared with its capabilities. They didn't raise the price of Pro.
What price increases?
I guess the Gemini price increase
3 replies →