Comment by ac29

> 35b-A3B runs with around 14 tok/s and partial offloading

FYI, this is what I am seeing for pure CPU inference so something is likely off with your setup.

Test setup is intel 13500 w/ 6 threads and 64GB DDR4 ram, a newer system should be much faster