Comment by 999900000999
2 hours ago
And hire 2 or 3 dev ops to keep it running ?
That another 400 to 700k.
It becomes your problem and not someone else’s. However, I don’t trust hosted LLMs for anything that needs to be private.
2 hours ago
And hire 2 or 3 dev ops to keep it running ?
That another 400 to 700k.
It becomes your problem and not someone else’s. However, I don’t trust hosted LLMs for anything that needs to be private.
Where do I sign up to get 200k/yr to keep one rack running? Sounds like an incredibly chill job
Apparently it’s going to take the 3 of us to do this, mate. Going to get so much reading done.
What you get is not what you cost.
40% overhead is quite typical, so you'd be looking at $120k/year. In the USA I'd consider that a competitive salary for an admin capable or keeping a $6M rack of specialized hardware running 24/7.
Yeah but you don't need two such people, or even one, dedicated to this single rack.
A company of the size that this is worthwhile for, probably has dedicated devops on staff already and can add this rack to the inventory with no additional staff.
2 replies →
> However, I don’t trust hosted LLMs for anything that needs to be private.
Why not? Do you trust AWS with things that need to be private?
More than I trust frontier labs. AWS doesn't need to recoup 9 digits USD of capex
So you can just use Bedrock?
> And hire 2 or 3 dev ops to keep it running
Not a devops but I'd say one full time is already too many.
Yes but zero is not enough and where do you get a fraction of a competent dev op from?
From the other dev-op work you’re doing.
2 replies →
There will be cloud/SaaS vendors who have lower cost of labor/capital due to automation and financing terms.
Having these models in the open caps the inference margin.
Just like that new jobs created by AI! Localized model maintainer/technician.
You’ll slap some training on existing technologists/infra/sysadmin folks and perhaps have a support contract for the edge cases (hardware troubleshooting and advanced replacement).
(managed an entire data center building with thousands of servers a lifetime ago with ~2-3 other people, it’s only gotten easier over the last two decades imho)
Just let it manage itself, what could go wrong! :)