Comment by chipdart
2 years ago
> NTFS is really horrible handling many small files.
To pile onto NTFS, it's performance is so notoriously bad that there are developer teams working on Windows projects that configure their build farm to do cross builds from Linux to Windows just to avoid the performance penalty.
As an anecdote, we had a really long build time for our pipeline (going back prob 15 years).
I argued for a Linux laptop, and the boss said, "OK, prove it. Here's two equivalent laptops, time it.".
Turns out there was zero difference, or negligible (Windows won), between compilation times. That has always annoyed me.
> Turns out there was zero difference, or negligible (Windows won), between compilation times.
I think there was something seriously flawed in your test. If you Google for a minute, you find multiple posts on how moving the same builds to Linux led to performance improvements in the range of 40% drops in build times.
Some anecdotes even compare doing the same builds in Ubunto with NTFS to see double-digit gains.
NTFS is notoriously awful in scenarios involving reading/writing many small projects. This is the bottleneck in Windows builds. There is a myriad of benchmarks documenting this problem.
Nowadays there are plenty of cross-platform projects to serve as benchmarks. Checking this can be as easy as checking out a project, start a full rebuild, and check how long it takes.
NTFS can be especially awful if you're used to Linux behaviour, port code over to Windows and expect the same performance.
Here's a talk on porting rustup to Windows: https://www.youtube.com/watch?v=qbKGw8MQ0i8
To begin with, it takes rustup 3m30s to install on Windows. After rejigging the code again and again away from its naive implementation which works fine on Linux, to perform sympathetically towards NTFS, it takes the same rustup 14s to install. That's quite a performance gain! But it needed a lot of changes to rustup, and to Windows itself.
1 reply →
That's survivor bias. Fewer people blog about unsuccessful initiatives.
2 replies →
It seems possible that their pipeline just had large files, or something like that.
Java/dotnet pipelines don't have Windows penalty. Both have dependencies as compiled modules zipped into large jar/dll. While even simple node project have 10s of thousands of files inside node_modules folder.
1 reply →
For the most part the two are pretty equivalent in performance, it's particularly constant blocking on small file IO where Windows falls way behind. Also a bit in the "tons of cores" and "tons of networking" categories but not as nearly as bad as small file IO.
Well yeah, if you avoid the OS (syscalls) as much as possible then all operating systems will have about the same performance.
Love that your boss was open to the idea but also wanted data to back it up. (imagine a slow clap)
A single datapoint from an experiment done by only one person you call it data? What's not to love...
1 reply →
Yep, was a good experience. I'm sure with fine tuning it (Linux) could've been better, but I ate that humble pie.
You can't have windows laptop without running a ton of Antivirus and other crapware .e.g MS store, HP/Dell diagnostics and MS Teams. If you factor that in, Linux will win everytime.
If your company has windows and Linux but only guards against threats on windows, then you may have a different problem.
I believe you can lock a Linux box down tighter than a windows box, but then you're trading compile times for other costs.
Not the case unfortunately; you can. I'm not defending Windows here (bitter hatred of Microsoft here, since they started charging 2k for the compilation tools back in the 90s).
In your particular corporate environment that might be the case, but not in this case, I had free run of a fresh install and no offensive AV there, and detuned to remove the crap.
Other posters have said certain optimizations (which I'm not sure would help, it was pure compilation, no large files that I'm aware of). Just saying, always good to keep an open mind.
Were all your tests pure unit tests? Then sure—the OS isn’t getting in the way (and if your Linux distribution didn’t set the correct cpu scheduler, windows could be faster!)
If, however, your tests make any filesystem calls or fork a child process, there’s slim chances that Linux doesn’t absolutely trounce Windows.
To throw in a morsel of anecdata: same laptop with a dual boot runs the PhotoStructure core test suite (some 11,000 tests which have a pleasing melange of system and integration tests, not just unit tests) in 8m30s on Windows 11. Almost all the same tests complete in 3m30s on Linux.
Seems weird to base the argument on compiler performance alone. I'd think that developer tools will be a much bigger factor for the time difference but that depends on what you are used to.
Anecdotally though, git is unbearably slow under Windows and compiles make all filesystem operations lag while I have never seen such problems under Linux.
Yeah sorry, I left out a lot of details for a quick anecdote. It was Java compilation, so the different JDK's may've made a difference (almost certainly). But I figured the better file system handling etc etc would've gone in favour of Linux, versus Windows.
In any case it wasn't much of a fun codebase. But I think a good lesson was, always test it, always measure. Not casting shade on either OS.
Do you remember which file system you used on Linux?
Didn't they introduced ReFS ("dev drives") to alleviate this?
Why wouldn't they work on improving NTFS performance instead?
Much of the cost is imposed by semantics and the security model; there's no silver bullet solution that can be imposed without turning it into a different filesystem with different semantics and/or on disk representation.
At one point they planned to replace it with a database filesystem, but that was too complicated and abandoned. That was probably the end of replacement work on NTFS. https://en.wikipedia.org/wiki/WinFS
WinFS was never designed as a replacement as it rode on top of NTFS. It was a fancy SQL Server database that exposed files as .NET Objects for API interaction.
Sans the API piece, think of it like storing blobs in SQL Server, just like SharePoint does.
I was lucky enough to play around with beta 1. Not much you could do with it, though.
> Why wouldn't they work on improving NTFS performance instead?
There are other old-school techniques which are far easier to implement and maintain, such as using RAM drives/partitions. Expensing 32GB of RAM is simpler than maintaining weird NTFS configurations.
Splitting your project into submodules/subpackages also helps amortize the impact of long build times. You can run multiple builds in parallel and then have a last build task to aggregate the all. Everyone can live with builds that take 5 minutes instead of 3.
Even Microsoft doesn't touch NTFS code (if they haven't lost it). All new file system features like new compression methods are implemented on layers above ntfs.
Because likely they aren’t a. allowed to, and/or b. have no file system code experience or understanding.