Comment by cyanydeez

https://github.com/peonist-ai/halogen-server is beating the pants off of anything I've tried.

38GB of vram resident. more tk/s, more prefill.