Comment by rishramanathan

Broadly, on the monitoring side, we’re more focused on evaluating the quality of the model’s outputs (is it violating your rules, handling specific subpopulations / edge cases correctly etc.). OpenLLMetry is more focussed on telemetry and tracing, whereas for us ‘monitoring’ is a means to running your tests on production data.

Openlayer’s also intended to be used on non-LLM use cases. Here are a few other ways we’re different:

1. Support for other ML task types

2. Includes a development mode for versioning and experimentation

3. Native slack and email alerts (openllmetry might integrate with other platforms that do that, but not sure)

4. Collaboration is deeply embedded into the product