Comment by tarpitt

2 days ago

I am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2

I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth.

>SSD Wear Warning

> Cold starts are heavy on random reads (~11 GB/token). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health.

Hmm, maybe a safe way to do this would be to make a separate partition for the model weights, and set them to read-only? Not sure how the page cache works, if it's like per partition or per disk. If it's per disk, maybe you could have a read-only data.iso formatted as a partition and mount it as a disk?

15 comments

tarpitt

Reply
vforno  2 days ago

I have a small laptop. If you have more disks available, you could really do some testing. When you have some benchmarks, submit a pull request or issue so we can maybe work on them. We are really happy for contribute!

  • tarpitt  2 days ago

    I have epyc 9654 ES and a 7900 XTX. I was running the numbers, and even if I maxxed out the ram to like 12x32 gig sticks, it would cost me thousands more and I could only run GLM-5.2 at a couple tokens per second at q3. So this project is very promising because it suggests I could get pretty high speed and this CPU/motherboard combination suggests I have a lot of pci bandwidth that is unused.

    I think another route might be looking at holding an even larger chunk of model weights in ram, and taking advantage of RAM<->GPU bandwidth, perhaps using a PCIe 5 GPU. This was my first thought since I have dedicated GPU.

    If you are using Laptop, you're looking at shared memory between the iGPU and CPU. I've also tried that route, but I have always been skeptical of killing flash with too many reads, it essentially uses SSD like it's a consumable item.

    I'm going to benchmark this right now with what I have and I'll get back to you on github.

    • halcyonblue  2 days ago

      At least for NVME, it is the write cycles that are limited. Read cycles are non-destructive and essentially unlimited.

    • throwdbaaway  1 day ago

      If you max out the ram, TG with q3 should be at least 10 t/s. And with dsa, it can still stay close to that number as the context grows.

valicord  2 days ago

> OS page cache can generate writes

Is this a hallucination? What am I missing? Why would heavy reads generate writes?

  • fallingbananna  2 days ago

    Good catch! Disk reads do generate writes to cache. But the cache itself is in RAM, not on disk. So it shouldn’t cause additional wear of SSD.

  • TacticalCoder  2 days ago

    > Is this a hallucination? What am I missing? Why would heavy reads generate writes?

    I take it heavy reads means more stuff goes into RAM, meaning other stuff has to be cached?

    I've got same question as GP: e.g. is there a way to set moderately fast consumer NVMe SSDs (I've got both a Samsung 990 Pro and a WD SN850X) in a complete read-only mode to prevent "wear"?