← Back to context

Comment by Havoc

1 day ago

Got a M5 air recently - my first dive into MacOS land so trying to figure this out too.

Seems essentially impossible to get:

* pytorch

* GPU acceleration

* VM/container like isolation

The virtio-gpu layer gets closest but seems to only pass through graphics GPU not compute GPU so no pytorch

7 comments

Havoc

Reply
plufz  1 day ago

I need this too, and looked quite a lot on it a year ago. I haven’t had time to check out the recent developments with Docker Model Runner (vllm-metal) or podman libkrun. Did neither of those work for you?

  • Havoc  1 day ago

    vllm-metal isn't GPU access but rather a openai compatible end point which I can already do via lm studio endpoint over network

    >podman libkrun

    Haven't tried it but research suggests its really shaky still. podman libkrun exposes vulkan while torch expects mps on macs. Sounds like one can force vulkan but that's apparently slow and beta-ish?

emmelaich  1 day ago

I got torch to run in a Cirruslabs Tart instance.

  • Havoc  20 hours ago

    By "Instance" do you mean their cloud platform?

    • emmelaich  11 hours ago

      Nah, just locally on my macair.

      TBF, I only got to the point that using device=mps_device didn't fail. I used Sonoma at the time and the image for the vm was ghcr.io/cirruslabs/macos-sequoia-xcode:16.2-beta-3. Python 3.12, as well, because torch didn't work with later versions.

         import torch
   mps_device = torch.device("mps")
   print('device is', mps_device)
   x = torch.ones(1, device=mps_device)
   print(x)