Comment by GodelNumbering
4 hours ago
I didn't have a local GPU, so I asked it to go out and find hardware. It found a google TPU v6e which seemed reasonably priced. I gave it my google api key. I told it to use TPU only when training and bring it down afterwards. That's about it.
> I gave it my google api key
This is the part where the narrator looks at the camera and says "Don't try this at home, kids!"
You’re absolutely right, I shouldn’t have rented a 200 GPU cluster for $35,000/hour. That’s on me.
[Search: Can I refund Google cloud?]
It looks like we’re not able to ask for a refund since we did actually use all of that compute intentionally.
Would you like me to write you a pleading email to send to the support team?
There’s a safer way to do this with nearly no added friction. Give it a read only API key. Then just ask it to write the API calls into a bash script and then read it and run it yourself. The agent can still inspect the live resources and diagnose and give you more commands to run. I do agree I wouldn’t give it create / write access.
Why? Isnt the API key scoped to a project and specifically made for this?
Are you confusing this with an OAuth token or something?
Until astra goes bonkers and use the tpu for days
1 reply →
I've done this sort of thing before but with Vast. Pre-deposited some money online, then let the LLM request and manage a training run on an allocation. Worked pretty well without risking bankruptcy.
What kind of observability did you have over this process? I’m interested in how my peers are operating these efforts.
On the cloud side, nothing valuable existed, so the training couldn't ruin anything it didn't create. On the laptop side, I usually ask the agents to create named scripts for everything it needs to access, then those local script directory is green-lit with approve all. For cost, I kept giving it new budget in the 20-30 dollar increments.
I had to intervene a few times. For instance, as smart as the models are said to be (Astra), it would copy the full training run, train on the server, pull every checkpoint to the local machine, then run tests, update. So, the bandwidth bill was as high as training bill for the first 6 hours. It could have simply tested each checkpoint on the server, saved time and money, didn't occur to it until I said.
Perhaps I wasn’t clear. What kind of instrumentation and alerting, if any, did you employ to keep an eye on it?
> I told it to use TPU only when training and bring it down afterwards.
I wouldn't put my house on it. Brave.
Neat!