Comment by 0cf8612b2e1e
3 hours ago
I saw someone’s excellent idea that if you have a slow system like this, you should communicate by email. It is no longer meant for realtime iteration, but more pointed questions for which there is more effort and time expected on both parties.
0.5t/s is still too slow even for email. For a moderately large inquiry (1MTok output, let's ignore the 4k context window limitation for now) it'll take the model around 23 days or uninterrupted execution to answer a single email.
Real world inquiries are gonna be much slower of course, but this setup is still too slow to do anything meaningfully useful I think.
I’m sure it’s possible, but I really struggle to think of an example that would result in a 1m token output.