If I call the same "get statistics" command over and over in a loop (with zero queries), or 100% the same invalid query (to test the error path performance), I believe we'd call that a micro-benchmark, despite involving a full database. It's a completely unrealistic artificial workload to test a particular type of operation.
The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.
In August 1990, the TPC approved its second benchmark, TPC-B. In contrast to TPC-A, TPC-B is not an OLTP benchmark. Rather, TPC-B can be looked at as a database stress test, characterized by:
Significant disk input/output
Moderate system and application execution time
Transaction integrity
TPC-B measures throughput in terms of how many transactions per second a system can perform. Because there are substantial differences between the two benchmarks (OLTP vs. database stress test), TPC-B results cannot be compared to TPC-A.
...
Transactions are submitted by programs all executing concurrently.
If I call the same "get statistics" command over and over in a loop (with zero queries), or 100% the same invalid query (to test the error path performance), I believe we'd call that a micro-benchmark, despite involving a full database. It's a completely unrealistic artificial workload to test a particular type of operation.
The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.
"get statistics" is not what TPC-B does. Nor the invalid queries nor ...
From https://www.tpc.org/tpcb/, a TPC-B workload that pgbench runs by default:
I think you missed the context of what I was responding to, which was about whether databases could even have micro-benchmarks.
You also missed the word "Obsolete" splattered all over the website you sent me, and the text that TPC-B was "Obsolete as of 6/6/95".
6 replies →