Comment by mhamann
3 months ago
Can you help me understand MetalRT a bit more? Based on the name, it sounds like something that's Apple-only (although, Apple basically co-opted the name Metal, which was traditionally more generic). Does or will MetalRT run on more platforms?
What about MetalRT's relationship to llama.cpp, onnx, MLX, transformers, etc? Is MetalRT a replacement for those? Designed to be compatible with a wide variety of model formats? Or are you just providing an abstraction on top of these?
MetalRT is metal only inference engine (we are making for other hardwares too). Think of it like SGLang or vLLM but for single batch inference on apple silicon. See this blogpost : https://www.runanywhere.ai/blog/metalrt-speech-fastest-stt-t...