Comment by brendangregg

10 months ago

From the docs: "pgbench is a simple program for running benchmark tests on PostgreSQL. It runs the same sequence of SQL commands over and over"

While it might call itself a benchmark, it behaves very microbenchmark-y.

The other numbers I and others have shared have been from actual production workloads. Not a simple program that tests same sequence of commands over and over.

12 comments

brendangregg

Reply
menaerus  10 months ago

While pgbench might be "simple" program, as in a test-runner, workloads that are run by it are far from it. It runs TPC-B by default but can also run your own arbitrary script that defines whatever the workload is? It also allows to run queries concurrently so I fail to understand the reasoning of it "being simple" or "microbenchmarkey". It's far from the truth I think.

weebull  10 months ago

Anything running a full database server is not micro.

  • brendangregg  10 months ago

    If I call the same "get statistics" command over and over in a loop (with zero queries), or 100% the same invalid query (to test the error path performance), I believe we'd call that a micro-benchmark, despite involving a full database. It's a completely unrealistic artificial workload to test a particular type of operation.

    The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.

    • menaerus  10 months ago

      "get statistics" is not what TPC-B does. Nor the invalid queries nor ...

      From https://www.tpc.org/tpcb/, a TPC-B workload that pgbench runs by default:

          In August 1990, the TPC approved its second benchmark, TPC-B. In contrast to TPC-A, TPC-B is not an OLTP benchmark. Rather, TPC-B can be looked at as a database stress test, characterized by:

      Significant disk input/output
      Moderate system and application execution time
      Transaction integrity

    TPC-B measures throughput in terms of how many transactions per second a system can perform. Because there are substantial differences between the two benchmarks (OLTP vs. database stress test), TPC-B results cannot be compared to TPC-A.

    ...

    Transactions are submitted by programs all executing concurrently.

      7 replies →

anarazel  10 months ago

The are loads of real world workloads that have similar patterns to pgbench, particularly read only pgbench.