Comment by kingnetart

(Disclosure: I work on inference at Antfly)

A little context on the inference and fine-tuning side of things at Antfly. Our model runtime is written in Zig, with no C++ bindings or dependencies. Over the last few months, we wrote our own CPU, Metal and CUDA kernels for the set of models that we support: Gemma4, Gliner2, Qwen3, Florence2, etc,

We’ve also created a JIT compiler for Metal/CUDA kernels, which should make it faster for us to add new kernel implementations in the future.

We're currently working on the infra to finetune (and deploy) Gliner2 and Gemma4 models directly in Antfly.

Happy to answer any questions about the inference runtime or fine-tuning work!