Comment by lewisjoe
1 day ago
It'd be great if it supports a wasm/web backend as well.
I bet a lot of trivial text capabilities (grammar checking, autocomplete, etc) will benefit from this rather than sending everything to a hosted model.
It's possible right now with onnx / transformers.js / tensorflow.js - but none of them are quite there yet in terms of efficiency. Given the target for microcontrollers, it'd be great to bring that efficiency to browsers as well.
If you need WASM, I think Candle is your current best bet: https://github.com/huggingface/candle
You can compile to wasm, I have done so via the XNNPACK backend - you might have to tweak the compilation settings and upgrade the XNNPACK submodule/patch some code. But this only supports CPU, not a WebGPU or WebGL backend.