← Back to context

Comment by Farmadupe

16 hours ago

@farlight assuming that you're the creator do you think you'd be able to rework the HTML/CSS? I'm sure you've got good data but speaking on behalf of my eyeballs, the results page is... hard to read!

> 2G of random garbage is written directly onto one member device (behind the filesystem's back, offset 1G — python injector; uutils dd mis-seeks on dm devices), caches dropped, then a full scrub: btrfs scrub -B, zpool scrub + wait, bcachefs scrub, md/lvm sync-action 'check' (which can only COUNT mismatches — no checksums to know which copy is right).

I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of? Can you confirm if any ofthe filesystems really came out of the other side in a usable state after scrubbing?

-----

> Trivial-op p99, idle (ms) # A trivial operation — one 4k write + fsync every 200ms (like a shell appending history or an editor updating its swap file) — run alone for 10s. p99 of the fsync completion

In fact, if it's OK for me to ask, are any of the metrics tht you used standard industry metrics? It looks like several of the tests are bypassing the kernel's page cache? -- which I worry may fall into the trap of "I modified the system to be unrepresentative of reality and then tested it".

----

> kernel 7.0.0-1012-azure

Can you confirm if you tested on a bare metal machine? were you the only tenant?

  • I answered in other thread how exactly integrity check looks like and why the name might be misleading. I'm also in the middle of updating descriptions to be more adequate. Thanks for pointing that out.

    Speaking about standard industry metrics.

    fio is an established tool, and throughput, IOPS, fsync latency, and percentiles are standard concepts. However, the exact job recipes and the composite “Overall Core” score are project-specific.

    The trivial-operation test is also custom: one 4 KiB write plus fsync every 200 ms. The idle window contains at most about 50 operations, so its p99 is effectively the slowest sample’s fio histogram bucket, not a statistically stable population percentile. It should be treated as a small-write durability-latency probe, not a universal application metric.

    But after all all tests are in the repository. If they need tweaks, changes I'm open to do so... I started from scratch and did whatever came to my mind. Some tests are added after my initial link here which went mostly unnoticed several weeks ago but I got some requests for more tests which I implemented.

    But to sum up. I want this test to be useful so feel free to open PRs with improvements. It's not like I've got some agenda. In fact I wrote here and there on the page that I'm counting on communities of various filesystems to provide improvements, changes etc to make their filesystem shining.

    This is personal project made when I realized that multiple-devices benchmarks were almost completely absent. Since I had not access to real hardware I decided to make at least initially everything based on GH runner with all the limitations that came with this approach. I tried to limit these limitations as far as I could. But feel free to submit bugreports, PRs, propositions for improvements.

  • > I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of?

    ZFS and btrfs were designed from the start to handle this, by using checksums on every piece of (meta)data and redundancy to return the same data as was stored to the kernel, and rewrite the bad data.

    I've tested my own machines running ZFS by random writes out of band from the filesystem/kernel and it has always found and fixed them.

what exactly would you like to improve?

  • I'd actually recommend some excellent books on the "philosophy" of data presentation: The first that comes to mind is "the visual display of quantitative information" by Edward Tufte seems to be freely available online, and the other one on my mind is "how charts lie" by Alberto Cairo (which doesn't seem to be freely accessible)

    But if it helps, just some "initial gut feel observations" from me:

    * It's definitely not possible to find issue with the the _sheer amount_ of results, but there's just far too much for a human to absorb, all presented at once

    * Overall text size is quite small, and difficult to read

    * The page doesn't make a strong statement of _what_ is under test: the first words are: "modern-fs-benchmark Multi-device CoW filesystems under workloads classic benchmarks skip" -- which defines the webpage in terms of what it is _not_, without stating what benchmarks are actually present.

    * The first line of teh page contains run statistics that probably eithre want to b at the bottom, or just don't need to be in the webpage at all: "latest run 2026-09-18 18:50:45 UTC, kernel 7.0.0-1012-azure, 593 runs recorded · 145 trend points shown"

    * A significant proportion of the free text is caveats. There's nothing wrong with being transparent about limitations, but they may be a sign that there might be alternative ways to present the data, or that the data may be flawed (depending on the caveat)

    * Theres several categories that I think have been invented for the purpose of collation, but I don't think are defined on the page. I think "Overall Core" and "Core I/O" aren't explained, which means by definition it's impossible for a reader to understand the score table.

    * And as we're all aware right now, current Claude models are currently struggling to write coherent English. There's several incoherent sentences on the page. It's a Claude issue.

    • It's not easy to show so many data and make everyone happy about the way it is presented. In fact I'm aiming more at engineers and trying to provide as much info as it is possible to be clear about methodology and everything around it. But raw data is in JSON files so you can always make PR for creating additional view aimed at philosophers and not engineers ;)

    • Speaking as someone who consumes this, I appreciate how it's laid out. As a developer, we can often see at a glance where the bottleneck is if we have enough data laid out - IOW, data overload for you is me feeling like a kid in a candy store.

      Sometimes there are ways to make things easier without dumbing them down, but way too many people conflate the two; I get nervous when non engineers say "I've studied this, it should be easy".

      2 replies →