← Back to context

Comment by hparadiz

4 hours ago

Here's my report running several different models on a dual Xeon with 256 GB of DDR4 and no GPU.

https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5...

4 comments

hparadiz

Reply
puzzlingcaptcha  4 hours ago

Have you tried with a single CPU to get rid of the NUMA penalty? I understand this likely means halving the memory but I am interested in how much of a difference it makes

  • trollbridge  3 hours ago

    I have (192GB machine with two CPUs), pretty much does the trick. It just runs some small models used for embedding, etc. and has those on one CPU / memory node and all the Docker containers on the other one.c

    • ryandrake  3 hours ago

      I have a dual xeon also, same as OP: Ivy Bridge + 128GB DRAM, and was never really able to get decent LLM performance out of it. So I ended up biting the bullet and adding a "budget tier" A4000 20GB GPU. Too bad all my DRAM is wasted now--not sure if there is a way to take advantage of lots of DRAM once you move over to having inference happening on the GPU.