Comment by losvedir
11 days ago
Something about having a very fast, very small harness written in C only to drive LLMs amuses me. Like putting a big ass spoiler on a minivan or something.
11 days ago
Something about having a very fast, very small harness written in C only to drive LLMs amuses me. Like putting a big ass spoiler on a minivan or something.
Well, you'd think so but given how slow and resource hungry the ones written in javascript are it makes a lot of sense.
Are they? I use pi and it doesn’t drain more than other software.
[flagged]
The LLM can run elsewhere, such as the cloud.
For the part that runs locally, efficiency might matter.