Comment by woadwarrior01

Judging by the code in the HF transformers repo[1], smaller dense versions of this model will most likely be released at some point. Hopefully, soon.

[1]: https://github.com/huggingface/transformers/tree/main/src/tr...