Comment by jopsen

It's never just running commands in a container.

Don't get me wrong, it's a fantastic primitive.

But eventually you need to conditionally run some tests (to save compute).

For some benchmarks you might have limited hardware, so you need to coalesce jobs, and only run every 5 or 10 commits. You might want to keep the hardware hot, but also the queue small. So ideally you want to coalesce dynamically.

You also want result reporting, comparisons to previous results. Oh, and since you're coalescing some jobs and doing others conditionally you'll need ways to manually trigger skipped jobs later, maybe bisect too.

It's when you need to economize your compute that CI can get really complex. Especially, if you have fragile benchmark that or flaky tests.

Yes, in theory you can enforce a culture that removes flaky tests, but doing so often requires tooling support -- statistics, etc.