Comment by girvo
1 day ago
I didn't spend that much, only $6500 AUD for a GB10 based Asus GX10 which is even slower than OPs, but I spent that because it makes for a great learning platform. Theres not much else that lets me fiddle with 128GB of RAM for my graphics processor, and it's quite lovely to be able to run things as long as I like without worrying about my cloud instance being shut down.
It's not financially a good idea: renting really does beat owning, and cloud beats both if you're only running inference on these machines. But I'm not just doing inference, and as a thing I can do silly stuff on to learn, it's hard to beat!
When you say you are not just doing inference, you mean you are also training your own llms? I am curious what other things can be done.
Fine tuning, and yeah training my own, experimenting with architectures and learning how it all works. Been a lot of fun
$6500 AUD can get you a good chunk of B200 time on any of the GPU neoclouds :)
Less than I expected, though! And I get to run this all through the night
I do still use Vast and Runpod for things too, but it’s much nicer to test a fine tuning run here to make sure I’m in the ballpark
I also did literally say “It's not financially a good idea, renting is better than owning” so I’m confused why I have two people telling me that
Also it’s just far more fun to play with something tangible to me :)
You could just rent a bare metal server with those specs
Yes I could, but that is annoying because of spot pricing and having my instance shut down, and it has fluctuating prices
It’s also annoying because then I need to make sure my little “lab” setup is well automated, and I’m lazy :)
Also, I literally said “ It's not financially a good idea” so I’m confused why you think I don’t know that.
Spot pricing and instance availability don’t apply to on metal hosting. You’d have your own machine dedicated to your own use only, at a locked in price.
1 reply →
[dead]