Comment by fooker
3 years ago
That's an interesting idea.
Are llama.cpp (and similar targets) made into standalone C++ source files without dependencies doing the heavy lifting?
3 years ago
That's an interesting idea.
Are llama.cpp (and similar targets) made into standalone C++ source files without dependencies doing the heavy lifting?
Yes, that's the idea of the project.
One can now compile against cuBLAS and other GPU libraries to help take some of the load, but the core of the project is indeed to run on the CPU and do all of the work without dependencies.
Is llama performance not basically bound by the memory hierarchy?
Bound by: yes. Currently restricted by: possibly not. For example the performance difference between clang 11 and 15 compiled llama.cpp is ~10% (I tested a while ago, can't remember the exact number)