Comment by docheinestages
2 days ago
What we really need is a breakthrough in inference or LLM architecture to allow running GLM-5.2-level models at the size of Qwen 3.6 27b or smaller on consumer devices like a 48GB Macbook Pro, and at least at 100 tokens/second. My hypothesis is that a smaller, less capable but faster model paired with a good harness can run for longer and brute force its way out to solve problems that the bigger models can one-shot.
That would be great during the winter months
Why not! And free hot showers during the summer. Just looking at the progress made since 2023, I don't think the LLM architecture we have today is the most efficient. We need creative game developers to start making LLMs.
Unfortunately I don't expect indie game devs can afford high spec hardware at current prices.
used to heat my dorm room with a Pentium D that I overclocked and pointed a small fan at. Could open a window in winter, turn off the room radiator, and keep it cozy.
perhaps at-home LLMs will bring me back to that. fun days of hacking and thermodynamics.
im more expecting the harness to be a literal LLM, Like how you put vibration dampeners on all kinds of mechanical structures