For this benchmark i 've been using OpenRouter. If you have enough RAM or you can ssh a cloud shell this intro https://memdoor.ai/docs/open-weight can work too. I have reached with the ssh door a Google Cloud Box where the 27B model ran with an L4 g2-standard-16. A inference engine server is setup easily with memdoor (https://github.com/guregodevo/janis) on any CPU or GPU box. Ideally i think we need a quorum of SSH boxes like this, distributed that you can ping with a mention such as @box:coder. It's WIP . Any suggestion for running a team of workers in remote machine is welcome. I'd like to find a cheap solution.
For this benchmark i 've been using OpenRouter. If you have enough RAM or you can ssh a cloud shell this intro https://memdoor.ai/docs/open-weight can work too. I have reached with the ssh door a Google Cloud Box where the 27B model ran with an L4 g2-standard-16. A inference engine server is setup easily with memdoor (https://github.com/guregodevo/janis) on any CPU or GPU box. Ideally i think we need a quorum of SSH boxes like this, distributed that you can ping with a mention such as @box:coder. It's WIP . Any suggestion for running a team of workers in remote machine is welcome. I'd like to find a cheap solution.