Comment by brendangregg

10 months ago

If I call the same "get statistics" command over and over in a loop (with zero queries), or 100% the same invalid query (to test the error path performance), I believe we'd call that a micro-benchmark, despite involving a full database. It's a completely unrealistic artificial workload to test a particular type of operation.

The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.

8 comments

brendangregg

menaerus 10 months ago

"get statistics" is not what TPC-B does. Nor the invalid queries nor ...

From https://www.tpc.org/tpcb/, a TPC-B workload that pgbench runs by default:

    In August 1990, the TPC approved its second benchmark, TPC-B. In contrast to TPC-A, TPC-B is not an OLTP benchmark. Rather, TPC-B can be looked at as a database stress test, characterized by:

      Significant disk input/output
      Moderate system and application execution time
      Transaction integrity

    TPC-B measures throughput in terms of how many transactions per second a system can perform. Because there are substantial differences between the two benchmarks (OLTP vs. database stress test), TPC-B results cannot be compared to TPC-A.

    ...

    Transactions are submitted by programs all executing concurrently.

brendangregg 10 months ago
I think you missed the context of what I was responding to, which was about whether databases could even have micro-benchmarks.
You also missed the word "Obsolete" splattered all over the website you sent me, and the text that TPC-B was "Obsolete as of 6/6/95".
- menaerus 10 months ago
  
  I don't think I have. I was only responding to the factually incorrect statement of yours that pgbench is a microbenchmark.
  > which was about whether databases could even have micro-benchmarks.
  No, this was an argument of yours that you pulled out out of nowhere. The topic very specifically was about the pgbench and not whether or not databases can have micro-benchmarks. Obvious answer is, yes, they can as any other software out there.
  I think that you kinda tried to imply that pgbench was one of such micro-benchmarks in disguise and which is why I c/p the description which proves that it is not.
  > You also missed the word "Obsolete"
  I did not since that was not the topic being discussed at all. And in a technical sense, it doesn't matter at all. pgbench still runs so it is very much "not obsolete".
  
  5 replies →