Comment by Veserv
6 hours ago
That just sounds like bad tracing implementations. A good tracing implementation should be able to drive gigabytes per second of trace logs to memory. If you are generating it slow enough to allow actual offload then you should be in the 1—10% range even if you are saturating your offload.
You should, of course, upper bound this overhead by switching to a full time travel debugging solution, thus tracing everything, when you get to the 10-30% range.
The only way you get to “majority” is if your trace implementation is slower than time travel debugging and provides less information, but then why choose something worse in every dimension.
I'm just reporting from the trenches here. I think you are suggesting that everyone is aware of and capable of using state-of-the-art (from 20 years ago) tracing schemes like XRay[1], when in reality they are not. Most projects would be well-served by any basic profiler but even profiling is apparently for wizards, because I've seen a lot of projects that will resort to manually annotating functions with OTel trace spans, which are ~millions of times more expensive than function calls. Even eBPF uprobe/uretprobe is 100x more expensive than XRay, at a minimum. HotSpot's JFR is like a miracle compared to what people suffer through to diagnose Rust+Tokio.
1: https://llvm.org/docs/XRay.html ... is there even a Rust analog to this?
Huh, it seems xray puts blank trampolines all over your binary? That sounds pretty nifty but I would expect it to be pretty language agnostic, ish? Adding support should be doable for Rust as well, right? Anyways, pretty nifty.
I am by no means an expert, but I've recently improved performance for some code and used tracy. They have rust bindings as well. It's pretty cool and it seems to be low overhead. Wonder if I can couple it with something like xray? Tracy is more the tracing library + tracing interpretations/aquisition tool.
Edit: apparently rust already supports xray natively on the nightly.
Not even an analog: https://doc.rust-lang.org/beta/unstable-book/compiler-flags/...
It's worth pointing out though that just tracing function calls isn't good enough for the kinds of stackless coroutines that run in async Rust tasks. You need a way of mapping between the async tasks and the compiler emitted traces.
afaik, C/C++ have the same problem.
The difference is nobody in the C++ community believes that a dominant asynchronous executor library exists, and there is not a pervasive belief that it would be helpful.
1 reply →