Comment by genthree

7 hours ago

Relatedly: Has anyone profiled the performance and reliability characteristics of rsyslogd (Linux and FreeBSD distributed syslogger, maybe other platforms too) in its mode where it’s shipping logs to a central node? I’ve configured and used it with relatively small (high single digit nodes, bursts of activity to a million or two requests per minute or so) set-ups but have wondered if there’s a reason it’s not a more common solution for distributed logging and tracing (yes it doesn’t solve the UI problem for those, but it does solve collecting your logs)

Like… has anyone done a Jepsen-like stress test on rsyslogd and shared the results? I’ve half-assedly looked before and not been able to find anything.

4 comments

genthree

Reply
ehostunreach  1 hour ago

Since this is an OTel-related submission, you could also use OTel collectors to collect and forward logs to a central OTel collector instance.

> yes it doesn’t solve the UI problem for those, but it does solve collecting your logs

I work for Netdata and over the last couple months, we've developed an external Netdata plugin that can ingest/index OTel logs [1]. The current implementation stores logs in systemd-compatible journal files and our visualization is effectively the same one someone would get when querying systemd journal logs [2]. i > Like… has anyone done a Jepsen-like stress test on rsyslogd and shared the results? I’ve half-assedly looked before and not been able to find anything.

I've not used rsyslogd specifically, but I don't see how you'd have any issues with the log volume you described.

[1] https://github.com/netdata/netdata/tree/master/src/crates/ne...

[2] https://learn.netdata.cloud/docs/logs/systemd-journal-logs/s...

jbaiter  5 hours ago

We're doing this with a few dozen GiBs of logs a day (rsylog -> central rsylog -> elasticsearch). It works reliably, but the config is an absolute nightmare, documentation is a mixed bag and troubleshooting often involves deep dives into the C code. We're planning to migrate to Alloy+Loki.