← Back to context

Comment by Farmadupe

18 hours ago

> 2G of random garbage is written directly onto one member device (behind the filesystem's back, offset 1G — python injector; uutils dd mis-seeks on dm devices), caches dropped, then a full scrub: btrfs scrub -B, zpool scrub + wait, bcachefs scrub, md/lvm sync-action 'check' (which can only COUNT mismatches — no checksums to know which copy is right).

I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of? Can you confirm if any ofthe filesystems really came out of the other side in a usable state after scrubbing?

-----

> Trivial-op p99, idle (ms) # A trivial operation — one 4k write + fsync every 200ms (like a shell appending history or an editor updating its swap file) — run alone for 10s. p99 of the fsync completion

In fact, if it's OK for me to ask, are any of the metrics tht you used standard industry metrics? It looks like several of the tests are bypassing the kernel's page cache? -- which I worry may fall into the trap of "I modified the system to be unrepresentative of reality and then tested it".

----

> kernel 7.0.0-1012-azure

Can you confirm if you tested on a bare metal machine? were you the only tenant?

4 comments

Farmadupe

Reply
fenio  4 hours ago

I answered in other thread how exactly integrity check looks like and why the name might be misleading. I'm also in the middle of updating descriptions to be more adequate. Thanks for pointing that out.

Speaking about standard industry metrics.

fio is an established tool, and throughput, IOPS, fsync latency, and percentiles are standard concepts. However, the exact job recipes and the composite “Overall Core” score are project-specific.

The trivial-operation test is also custom: one 4 KiB write plus fsync every 200 ms. The idle window contains at most about 50 operations, so its p99 is effectively the slowest sample’s fio histogram bucket, not a statistically stable population percentile. It should be treated as a small-write durability-latency probe, not a universal application metric.

But after all all tests are in the repository. If they need tweaks, changes I'm open to do so... I started from scratch and did whatever came to my mind. Some tests are added after my initial link here which went mostly unnoticed several weeks ago but I got some requests for more tests which I implemented.

But to sum up. I want this test to be useful so feel free to open PRs with improvements. It's not like I've got some agenda. In fact I wrote here and there on the page that I'm counting on communities of various filesystems to provide improvements, changes etc to make their filesystem shining.

This is personal project made when I realized that multiple-devices benchmarks were almost completely absent. Since I had not access to real hardware I decided to make at least initially everything based on GH runner with all the limitations that came with this approach. I tried to limit these limitations as far as I could. But feel free to submit bugreports, PRs, propositions for improvements.

matja  16 hours ago

> I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of?

ZFS and btrfs were designed from the start to handle this, by using checksums on every piece of (meta)data and redundancy to return the same data as was stored to the kernel, and rewrite the bad data.

I've tested my own machines running ZFS by random writes out of band from the filesystem/kernel and it has always found and fixed them.