Comment by exabrial
7 hours ago
https://github.com/exabrial/petrify
Petrify is a machine learning model compiler for the the JVM. It reads your model from an ONNX or other model format, walks the Trees or Linear models, and encodes the model in equivalent JVM bytecode as a stateless class you can invoke.
This differs from every other ONNX Runtime that I know of, which are essentially interpreters. The ONNX Runtimes are also huge (90+mb!?!), JNI, and drag gargantuan dependencies!
This just compiles your models to native bytecode. Much simpler and you end up with 0 dependencies! (you need one interface technically, but I digress).
This is interesting. I’ve been working with ONNX models and compiling custom WASM runtime builds based on the model operators, cutting down on edge deployments.
Do you have any benchmarks?
not... yet! Speed actually was a byproduct hilariously. Compiled models are definitely lightning fast, likely the fastest they could ever be on the JVM, because the tree is directly encoded as bytecode; represented lots of Opcode.IFGT-like comparisons. The JVM's JIT Compiler will have a blast with these code paths.
Petrify will also be order of magnitude kinder to your Garbage Collector, which will increase performance in high-throughput situations. You're also not loading 10 gazillion classes, as your models are directly represented as a first-class Java Class.
The real goal here was the getting rid of dependencies! While thankful for the incredible (and free) work of the authors of the onnxruntime for Java, the primary onnxruntime jar a boat anchor; weighing is 90mb+ just by itself, not counting any of its dependencies.
Once you compile your models with Petrify, you have exactly one 6.9kb jar as a dependency essentially just carries the Fossil interface as an entry point to call your model. I licensed that jar ASL2.0 for maximum compatibility in a corporate environment.