Comment by nine_k
4 hours ago
I suppose the insane speed is due to this:
> TurboKV's persisted Bloom-filter format uses hardware AES.
Also, built-in LZ4 compression.
I would expect SIMD to be used for scans.
4 hours ago
I suppose the insane speed is due to this:
> TurboKV's persisted Bloom-filter format uses hardware AES.
Also, built-in LZ4 compression.
I would expect SIMD to be used for scans.
I assume this is for hashing. I've seen several hashing algorithms turn to hardware AES instructions before, but I haven't seen any evidence that this technique outperforms state-of-the-art hashes like RapidHash (https://github.com/Nicoshev/rapidhash) in either quality or speed.
Those help but the main write speed gain is the WAL, that uses preallocated mmap segments to avoid a write(2) per durable mutation while preserving crash recovery. AES hashing mainly helps Bloom filter point lookups and LZ4 mainly helps SSTable I/O. Scans benefit indirectly, but don’t yet use a custom SIMD merge loop.
I said elsewhere this doesn't survive a power loss.
paranoid() does survive power loss.
While it’s important to make this explicit, at what point do we just assume a high-reliability UPS is table stakes?
Of course, if you need SIL2 type reliability then you need to assume any given hardware component can spontaneously combust and become a total loss, at which point the data loss caused by a power cut is a rounding error.
2 replies →
Iirc that’s how badger handles the WAL as well