← Back to context

Comment by atombender

6 months ago

Thanks! Very familiar with ClickHouse, but can logs then be ingested into CH without going through HyperDX? Doesn't HyperDX require a specific schema that the Vector pipeline would have to adapt the payloads to?

5 comments

atombender

Reply
mikeshi42  6 months ago

Nope! We're virtually schema agnostic, you can map your custom schema to observability concepts (ex. the SQL expression for TraceID, either a column or a full function/expression will work).

We don't have any lock in to our ingestion pipeline or schema. Of course we optimize a lot for the OTel path, but it works perfectly fine without it too.

  • atombender  6 months ago

    Very happy with CH, but I'm sadly disappointed with HyperDX:

    * Not possible to reorder columns?

    * Not possible to wrap cells?

    * The doesn't seem to be a concept of "field popularity" as Kibana has (where you can also "pin" fields)?

    * The log view's chart is very simplistic. No breakdown. Time selector is very primitive. Look at Google Cloud's log view if you want to see something good here.

    * No field value autocompletion when using the query builder?

    * Live view is annoying as hell. It scrolls to the top every few seconds even when there is no need data.

    * Chart view is nearly useless. I tried to create a chart showing two time series calculations, average and P95 of a metric, and it doesn't draw it correctly, and the series get messed up, and overall I think it's not usable. Happy to explain further, but even a cursory test should reproduce this. Fortunately Grafana can access CH data and do a much better job here.

    * The drill-down sidebar doesn't seem to have any idea what's important or not, so some fields I will never want to filter on are high up in the list and others are further down. I can't rearrange the fields?

    * Lots of other nuisances.

    Overall I found it much, much weaker than Kibana for both logs and charts, and Kibana is already pretty atrocious at both, so.

    I can probably live with it because I'm desperate to replace Elastic with CH, but I think I will kiss the functionality of the Kibana UI.

    • mikeshi42  6 months ago

      This is really helpful feedback, a few notes:

      - Reorder columns: You should be able to do so by modifying the order of the SELECT statement at the top.

      - Wrapping: totally agree - this is something we're adding in, it's on the near term todos.

      - Pin fields: you're totally right, you can pin values but that doesn't prioritize the filter in general. I'm getting this one in the queue.

      - Time picker: I hear you with Google Cloud's time picker, they did indeed make a really nice one. Though I'm curious to hear more about the breakdown, is it wanting to customize how that chart is grouped beyond log level?

      - Autocomplete: We had a regression if you're on v2.0.0 which is what I suspect you're hitting. If you're on v2.0.1 lucene should have field value autocomplete. We don't have it yet for SQL.

      - Live View: Can you clarify what is scrolling? Or is this about disabling live?

      - Chart View: I just tried really quick and it seems okay to me, do you mind sharing more details?

      - Sidebar: Is this related to field popularity to pin fields or something else?

      This feedback is all super helpful - you can probably tell we're still early in building out the perfect experience. I'd love it if we could dive in deeper either on our discord (https://hyperdx.io/discord) or email: michael.shi@clickhouse.com. Either way this has been exceptionally helpful :)

      1 reply →