Comment by etaioinshrdlu
1 year ago
My main question is why observability data needs (or benefits from) a tailor-made database instead of a general purpose one. In 2025, anyone working on observability who told me they have to build their own database, I would be very suspicious!
Datadog always builds their own event store: https://www.datadoghq.com/blog/engineering/introducing-husky...
It may not be named "database" but actually take the place of a database.
Observability vendors will try to store logs with ElasticSearch and later find it over expensive and has weak support for archiving cold data. Data Warehouse solution requires a complex ETL pipeline and can be awkward when handling log data (semi-structured data).
That said, if you're building an observability solution for a single company, I'd totally agree to start with single node PG with backup, and only consider other solution when data and query workload grow.
In 2025 I'd consider starting with clickhouse instead, if you're going the DIY route
Not even limited to general purpose ones, there are existing tailor made databases for observability. Maybe somewhere on that page, they explain why this one is better.