Broadly, on the monitoring side, we’re more focused on evaluating the quality of the model’s outputs (is it violating your rules, handling specific subpopulations / edge cases correctly etc.). OpenLLMetry is more focussed on telemetry and tracing, whereas for us ‘monitoring’ is a means to running your tests on production data.
Openlayer’s also intended to be used on non-LLM use cases. Here are a few other ways we’re different:
1. Support for other ML task types
2. Includes a development mode for versioning and experimentation
3. Native slack and email alerts (openllmetry might integrate with other platforms that do that, but not sure)
4. Collaboration is deeply embedded into the product
Traceloop's landing page is all about model quality, not metrics. Their open source OpenLLMetry is the metrics part and hooks into the OpenTelemetry ecosystem. There should be no issue with getting alerts via the ecosystem, it's promanent on their pages.
I think the target personas is different. While they might have the same capabilities, but the job-to-be-done is different.
openllmetry is focus on engineers, who wants to use this as more of a piping solution and it sits on top of opentelemtry. While opentelemetry is a popular solution. It is just applying a solution to a new problem.
OpenLayers to me is thinking from the ML/AI problems from ground up and while serving the data scientists and probably prompt engineers.
1. OpenLayer does not say metrics or monitoring to me
2. OpenLLMetry builds on OpenTelemetry, which it very much reminds me of as a name. It's also a much easier add-on to our existing stack. I don't want to have to log into some company's website to view metrics for a single part of my stack when trying to understand why things are not working as expected.
3. OpenLLMetry is open core, which is what devs desire. Who is really using closed source software in this space now (the logmon space, not ai, though both are largely chasing after open dreams)
Broadly, on the monitoring side, we’re more focused on evaluating the quality of the model’s outputs (is it violating your rules, handling specific subpopulations / edge cases correctly etc.). OpenLLMetry is more focussed on telemetry and tracing, whereas for us ‘monitoring’ is a means to running your tests on production data.
Openlayer’s also intended to be used on non-LLM use cases. Here are a few other ways we’re different:
1. Support for other ML task types
2. Includes a development mode for versioning and experimentation
3. Native slack and email alerts (openllmetry might integrate with other platforms that do that, but not sure)
4. Collaboration is deeply embedded into the product
Traceloop's landing page is all about model quality, not metrics. Their open source OpenLLMetry is the metrics part and hooks into the OpenTelemetry ecosystem. There should be no issue with getting alerts via the ecosystem, it's promanent on their pages.
https://www.traceloop.com/
I think the target personas is different. While they might have the same capabilities, but the job-to-be-done is different.
openllmetry is focus on engineers, who wants to use this as more of a piping solution and it sits on top of opentelemtry. While opentelemetry is a popular solution. It is just applying a solution to a new problem.
OpenLayers to me is thinking from the ML/AI problems from ground up and while serving the data scientists and probably prompt engineers.
And Langfuse - https://news.ycombinator.com/item?id=37310070
For one thing, the name is certainly better than the latter's lol
I find OpenLLMetry to be better.
1. OpenLayer does not say metrics or monitoring to me
2. OpenLLMetry builds on OpenTelemetry, which it very much reminds me of as a name. It's also a much easier add-on to our existing stack. I don't want to have to log into some company's website to view metrics for a single part of my stack when trying to understand why things are not working as expected.
3. OpenLLMetry is open core, which is what devs desire. Who is really using closed source software in this space now (the logmon space, not ai, though both are largely chasing after open dreams)