{"id":44978062,"user":"decide1000","time":1755810201,"time_ago":"6 months ago","type":"comment","content":"

I use it on a 24gb gpu Tesla P40. Very happy with the result.","comments":[{"id":44978305,"user":"hkt","time":1755811700,"time_ago":"6 months ago","type":"comment","content":"

Out of interest, roughly how many tokens per second do you get on that?","comments":[{"id":44978371,"user":"edude03","time":1755812135,"time_ago":"6 months ago","type":"comment","content":"

Like 4. Definitely single digit. The P40s are slow af","comments":[{"id":44979987,"user":"coolspot","time":1755824189,"time_ago":"6 months ago","type":"comment","content":"

P40 has memory bandwidth of 346GB/s which means it should be able to do around 14+ t/s running a 24 GB model+context.","comments":[{"id":45043333,"user":"edude03","time":1756319908,"time_ago":"5 months ago","type":"comment","content":"

Not sure why I got downvoted - literally the first result (for me) says the best result[0] is 11t/s at Q3. Everything else is single digits, like 2-8t/s. Also considering that its not supported anymore[1] (It's Compute Capability is 6.1, not supported by cuda anymore) and it's power draw, I'd highly recommend anyone interested in ML stay far away from it - even if its all you can afford.

While the memory bandwidth is decent, you do actually need to do matmuls and other compute operations for LLMs, which again its pretty slow at

[0]: https://old.reddit.com/r/LocalLLaMA/comments/1dcdit2/p40_ben...\n[1]: https://developer.nvidia.com/cuda-gpus","comments":[],"comments_count":0,"level":3,"url":"item?id=45043333"}],"comments_count":1,"level":2,"url":"item?id=44979987"}],"comments_count":2,"level":1,"url":"item?id=44978371"}],"comments_count":3,"level":0,"url":"item?id=44978305"}],"comments_count":4,"url":"item?id=44978062"}

Comment by decide1000