Comment by mohsen1
9 hours ago
it is basically a “performance regression detector” It looks at a time series of benchmark results (e.g. test runtime, latency, memory usage across commits) and tries to answer one question: did something actually change, or is this just noise?
in a performance critical project I am working on, I had a precommit hook to run microbenchmarks to avoid perf regressions but noise was a real issue. Have to to try it to be sure if it can help but seems like a solution to a problem I had.
If your benchmarks are fast enough to run in pre-commit you might not need a time series analysis. Maybe you can just run an intensive A/B test between HEAD and HEAD^.
You can't just set a threshold coz your environment will drift but if you figure out the number of iterations needed to achieve statistical significance for the magnitude of changes you're trying to catch, then you might be able to just run a before/after then do a bootstrap [0] comparison to evaluate probability of a change.
[0] https://en.wikipedia.org/wiki/Bootstrapping_(statistics)