Comment by kelseydh
3 months ago
I recently did performance testing of Tigerbeetle for a financial transactions company. The key thing to understand about Tigerbeetle's speed is that it achieves very high speeds through batching transactions.
----
In our testing:
For batch transactions, Tigerbeetle delivered truly impressive speeds: ~250,000 writes/sec.
For processing transactions one-by-one individually, we found a large slowdown: ~105 writes/sec.
This is much slower than PostgreSQL, which row updates at ~5495 sec. (However, in practice PostgreSQL row updates will be way lower in real world OLTP workloads due to hot fee accounts and aggregate accounts for sub-accounts.)
One way to keep those faster speeds in Tigerbeetle for real-time workloads is microbatching incoming real-time transactions to Tigerbeetle at an interval of every second or lower, to take advantage of Tigerbeetle's blazing fast batch processing speeds. Nonetheless, this remains an important caveat to understand about its speed.
Hi! Rafael from TigerBeetle here!
> One way to keep those faster speeds in Tigerbeetle for real-time workloads is microbatching incoming real-time transactions to Tigerbeetle at an interval of every second or lower, to take advantage of Tigerbeetle's blazing fast batch processing speeds.
We don’t recommend artificially holding transfers just for batching purposes. René actually had to implement a batching worker API to work around a limitation in Python’s FastAPI, which handled requests per process, and he’s been very clear in suggesting that such would be better reimplemented in Go.
Unlike most connection-oriented database clients, the TigerBeetle client doesn’t use a connection pool, because there’s no concept of a “connection” in TigerBeetle’s VSR protocol.
This means that, although you can create multiple client instances, in practice less is better. You should have a single long-lived client instance per process, shared across tasks, coroutines, or threads (think of a web server handling many concurrent requests).
In such a scenario, the client can efficiently pack multiple events into the same request, while your application logic focuses solely on business-event-oriented chains of transfers. Typically, each business event involves only a handful of transfers, which isn't a problem of underutilization, as they'll be submitted together with other concurrent events as soon as possible.
However, if you’re dealing with a non-concurrent workload, for example, a batch process that bills thousands of customers for their monthly invoices, then you can simply submit all transfers at once.
Joran from TigerBeetle!
> For processing transactions one-by-one individually
If you're artificially restricting the load going into TigerBeetle, by sending transactions in one-by-one individually, then I think predictable latency (and not TPS) would be a better metric.
For example, TB's multi-region/multi-AZ fault-tolerance will work around gray failure (fail slow of hardware, as opposed to fail stop) in your network links or SSDs. You're also getting significantly stronger durability guarantees with TB [0][1].
It sounds like you were benchmarking on EBS? We recommend NVMe. We have customers running extremely tight 1 second SLAs, seeing microsecond latencies, even for one at a time workloads. Before TB, they were bottlenecking on PG. After TB, they saturated their central bank limit.
I would also be curious to what scale you tested? We test TB to literally 100 billion transactions. It's going to be incredibly hard to replicate that with PG's storage engine. PG is a great string DBMS but it's simply not optimized for integers the way TB is. Granted, your scale likely won't require it, but if you're comparing TPS then you should at least compare sustained scale.
There's also the safety factor of trying to reimplement TB's debit/credit primitives over PG to consider. Rolling it yourself. For example, did you change PG's defaults away from Read-Committed to Serializable and enable checksums in your benchmarks? (PG's checksums, even if you enable them, are still not going to protect you from misdirected I/O like the recent XFS bug.) Even the business logic is deceptively hard, there are thousands of lines of complicated state machine code, and we've invested literally millions into testing and audits.
Finally, it's important that your architecture as a whole, the gateways around TB, designs for concurrency first class, and isn't "one at a time", or TigerBeetle is probably not going to be your bottleneck.
[0] https://www.youtube.com/watch?v=_jfOk4L7CiY
[1] https://jepsen.io/analyses/tigerbeetle-0.16.11
Doesn't the Tigerbeetle client automatically batch requests?
We didn't observe any automatic batching when testing Tigerbeetle with their Go client. I think we initiated a new Go client for every new transaction when benchmarking, which is typically how one uses such a client in app code. This follows with our other complaint: it handles so little you will have to roll a lot of custom logic around it to batch realtime transactions quickly.
I'm a bit worried you think instantiating a new client for every request is common practice. If you did that to Postgres or MySQL clients, you would also have degradation in performance.
PHP has created mysqli or PDO to deal with this specifically because of the known issues of it being expensive to recreate client connects per request
5 replies →
Interesting, I thought I had heard that this is automatically done, but I guess it's only through concurrent tasks/threads. It is still necessary to batch in application code.
https://docs.tigerbeetle.com/coding/clients/go/#batching
But nonetheless, it seems weird to test it with singular queries, because Tigerbeetle's whole point is shoving 8,189 items into the DB as fast as possible. So if you populate that buffer with only one item your're throwing away all that space and efficiency.
2 replies →
> We didn't observe any automatic batching when testing Tigerbeetle with their Go client.
This is not accurate. All TigerBeetle's clients also auto batch under the hood, which you can verify from the docs [0] and the source [1], provided your application has at least some concurrency.
> I think we initiated a new Go client for every new transaction when benchmarking
The docs are careful to warn that you shouldn't be throwing away your client like this after each request:
Again, I would double check that your architecture is not accidentally serializing everything. You should be running multiple gateways and they should each be able to handle concurrent user requests. The gold standard to aim for here is a stateless layer of API servers around TigerBeetle, and then you should be able to push pretty good load.
[0] https://docs.tigerbeetle.com/coding/requests/#automatic-batc...
[1] The core batching logic powering all language clients: https://github.com/tigerbeetle/tigerbeetle/blob/main/src/cli...
4 replies →
Did the company end up using it?
We didn't rule out using Tigerbeetle, but the drop in non-batch performance was disappointing and a reason we haven't prioritised switching our transaction ledger from PostgreSQL to Tigerbeetle.
There was also poor Ruby support for Tigerbeetle at the time, but that has improved recently and there is now a (3rd party) Ruby client: https://github.com/antstorm/tigerbeetle-ruby/
I think the drop in non-batch performance was more a function of the PoC than of TB. Would love to see what our team could do for you here! Feel free to reach out to peter@tigerbeetle.com