Comment by hosh
2 days ago
I liked Otel for traces and maybe logging -- but I think the Otel metrics is over-engineered.
Does ClickStack have a way to ingest statsd data, preferably with Datadog extensions (which adds tagging)?
Does ClickStack offer correlations across traces, logging, and metrics via unified service tagging? Does the UI offer the ability to link to related traces, logging, and metrics?
Why does the Elixir sdk use the hyperdx library instead of the otel library?
Are Notebooks in the roadmap?
Great questions!
OTel Metrics: I get it, it's specified as almost a superset of everyone's favorite metric standards with config for push/pull, monotonic vs delta, exponential/"native" histograms, etc. I have my preferences as well which would be a subset of the standard but I get why a unifying standard needed to be flexible.
Statsd: The great thing about the OTel collector is that it allows ingesting a variety of different data formats, so you can take in statsd and output OTel or write directly to ClickHouse: https://github.com/open-telemetry/opentelemetry-collector-co...
We correlate across trace/span id as well as resource attributes. The correlation across logs/traces with span/trace id is a pretty well worn path across our product. Metrics to the rest is natively done via resource attributes and we primarily expose correlation for K8s-based workloads with more to come. We don't do exemplars _yet_ to solve the more generic correlation case for metrics (though I don't think statsd can transmit exemplars)
Elixir: We try to do our best to support wherever our users are, the OTel SDK and ours have continued to change in parallel over time - we'll want to likely re-evaluate if we should start pointing towards the base OTel SDK for Elixir. We've been pretty early on the OTel SDK side across the board so things continue to evolve, for example our Deno OTel integration came out I think over a year before Deno officially launched one with native HyperDX documentation <3
Notebooks: Yes, it should land in an experimental state shortly, stay tuned :) There's a lot of exciting workflows we're looking to unlock with notebooks as well. If you have any thoughts in this direction, please let me know. I'd love to get more user input ahead of the first release.
Thank you. I saw a different thread about Otel statsd receiver, so that works out better. The last time I had looked into it, the otel metrics specs were very complex.
I think this is enough features for me to seriously take a look at it as a Datadog alternative.
> but I think the Otel metrics is over-engineered.
What about OTel metrics is difficult?
You can set up receivers for other metrics sources like stasd or even the DD agent, so there's no need to immediately replace your metrics stack.
One critical problem for me: no support for raw metrics.
Sometimes, you just want to export ALL of your metrics to the server and let it deal with histograms, aggregation, etc.
Another annoyance is the API, you can't just put "metrics.AddMeasurement('my_metric', 11)", you have to create a `Meter` (which also requires a library name), and then use it.
My foray into otel with aws lambda was not a success (about 6 months ago). Many of my issues were with the prom remote writer that I had to use. The extension was not reliable. Queue errors were common in the remote writer. Interop with Prometheus labels was bad. And the various config around delta and non-delta metrics was a bit of a mess. The stack I was using at least didn’t support exponential histograms. Got it to work mostly after days of fiddling but never reliably. Ripped it out and was happier. Maybe a pure OTEL stack would have been a much better experience than needing the prom remote writer - which I’d like to try in the future.
I’d certainly appreciate hearing success stories of OTEL + serverless.