NTFS is really horrible handling many small files. When compiling/watching node modules (easily 10-100k files), we've seen a 10x size difference internally (same hardware, just different OSes). At some point that meant a compile time difference of 10-30 sec vs 6-10 min. Not fun.
NTFS is perfectly fine at handling small files and performs on-par with other modern file systems.
The issue is Defender in sync mode/other AV/other file system filters.
DevDrive as noted by default uses an async scanning technique as well as ReFS. ReFS will suffer the exact same performance issues with Defender (or other AV/other file system filters) doing its thing when running in sync mode, which it does by default for ReFS-formatted drives in Windows Server.
> Except for CloseHandle(). These calls were often taking 1-10+ milliseconds to complete.
> While I didn't realize it at the time, the cause for this was/is Windows Defender. Windows Defender (and other anti-virus / scanning software) typically work on Windows by installing what's called a filesystem filter driver.
This doesn't take away from your point that _it is slow_, but the reasons are not due to the file system in use.
>The issue is Defender in sync mode/other AV/other file system filters.
I've had folders take a full minute to open on an SSD.
It got to the point where I went to open the folder, it started loading. I needed the file quickly, so I searched for it online, found it, and opened it before windows finished loading that folder for me.
After exempting that folder from Windows Defender the folder loads instantly. For the life of me I cannot understand why Defender blocks Explorer.
Technically, they're because of the filesystem in use: it's providing the APIs these garbage-ware utilize... which causes the performance issues ( ◠ ‿ ・ ) —
I don't know if async, AV, NTFS, or what, but some years ago we could demonstrate reliably that running a "big" maven compile on a lot of .java files was faster in Linux than Windows.
The kicker... Linux was running inside a VirtualBox VM inside the very same Windows host.
This could also be some variance in the `javac` command between OS's, granted.
> NTFS is really horrible handling many small files.
To pile onto NTFS, it's performance is so notoriously bad that there are developer teams working on Windows projects that configure their build farm to do cross builds from Linux to Windows just to avoid the performance penalty.
It's more a case of Linux programs not being written to work around the performance issues of Windows filesystems + layers above them. NTFS doesn't offer magical featurs that fix the performance.
That, and continuous scanning of all file IO corporations love so much can 2x-10x that time still. Dev drive is excluded from Defender scrutiny by default because of that.
I’ve benchmarked deleting files (around ~65,000 small node_modules sort of files) and it takes 40 seconds through Explorer, 20 seconds with rd, and roughly a second inside WSL2 (cloned to the VM’s ext4 virtual hard drive).
Their "windows dev drive" setup addresses this. I haven't tested it myself but I saw a couple of inexpertly executed tests showing significant performance gains. I honestly have no idea if my compile times are quicker.
But the primary improvement comes from async A/V, not the file system.
ReFS offers other improvements such as CoW and instant file initialization which can benefit developers. From this standpoint, it is the correct choice over NTFS.
NTFS is really horrible handling many small files. When compiling/watching node modules (easily 10-100k files), we've seen a 10x size difference internally (same hardware, just different OSes). At some point that meant a compile time difference of 10-30 sec vs 6-10 min. Not fun.
NTFS is perfectly fine at handling small files and performs on-par with other modern file systems.
The issue is Defender in sync mode/other AV/other file system filters.
DevDrive as noted by default uses an async scanning technique as well as ReFS. ReFS will suffer the exact same performance issues with Defender (or other AV/other file system filters) doing its thing when running in sync mode, which it does by default for ReFS-formatted drives in Windows Server.
https://news.ycombinator.com/item?id=26737521
> Except for CloseHandle(). These calls were often taking 1-10+ milliseconds to complete.
> While I didn't realize it at the time, the cause for this was/is Windows Defender. Windows Defender (and other anti-virus / scanning software) typically work on Windows by installing what's called a filesystem filter driver.
This doesn't take away from your point that _it is slow_, but the reasons are not due to the file system in use.
>The issue is Defender in sync mode/other AV/other file system filters.
I've had folders take a full minute to open on an SSD.
It got to the point where I went to open the folder, it started loading. I needed the file quickly, so I searched for it online, found it, and opened it before windows finished loading that folder for me.
After exempting that folder from Windows Defender the folder loads instantly. For the life of me I cannot understand why Defender blocks Explorer.
2 replies →
Technically, they're because of the filesystem in use: it's providing the APIs these garbage-ware utilize... which causes the performance issues ( ◠ ‿ ・ ) —
5 replies →
I don't know if async, AV, NTFS, or what, but some years ago we could demonstrate reliably that running a "big" maven compile on a lot of .java files was faster in Linux than Windows.
The kicker... Linux was running inside a VirtualBox VM inside the very same Windows host.
This could also be some variance in the `javac` command between OS's, granted.
Doesn't it (Windows VFS layer) also lack the equivalent of dentry cache, making all metadata lookups slow?
3 replies →
> NTFS is really horrible handling many small files.
To pile onto NTFS, it's performance is so notoriously bad that there are developer teams working on Windows projects that configure their build farm to do cross builds from Linux to Windows just to avoid the performance penalty.
As an anecdote, we had a really long build time for our pipeline (going back prob 15 years).
I argued for a Linux laptop, and the boss said, "OK, prove it. Here's two equivalent laptops, time it.".
Turns out there was zero difference, or negligible (Windows won), between compilation times. That has always annoyed me.
22 replies →
Didn't they introduced ReFS ("dev drives") to alleviate this?
Why wouldn't they work on improving NTFS performance instead?
5 replies →
That may be due to a combination of Malware detection + most unix programs not really written to take advantage of the features NTFS has to offer
This is a great talk on the topic https://youtu.be/qbKGw8MQ0i8?si=rh6WJ3DV0jDZLddn
It's more a case of Linux programs not being written to work around the performance issues of Windows filesystems + layers above them. NTFS doesn't offer magical featurs that fix the performance.
Must be why windows 11 added that dev drive feature
That, and continuous scanning of all file IO corporations love so much can 2x-10x that time still. Dev drive is excluded from Defender scrutiny by default because of that.
1 reply →
It’s somewhat more complex than “NTFS is slow”. Here’s a good explanation: https://github.com/Microsoft/WSL/issues/873#issuecomment-425...
I’ve benchmarked deleting files (around ~65,000 small node_modules sort of files) and it takes 40 seconds through Explorer, 20 seconds with rd, and roughly a second inside WSL2 (cloned to the VM’s ext4 virtual hard drive).
Why not use https://github.com/bobranten/Ext4Fsd ??
So we should put node_modules into SQLite database?
Honestly, maybe? Would be a fun project to look at at least.
Their "windows dev drive" setup addresses this. I haven't tested it myself but I saw a couple of inexpertly executed tests showing significant performance gains. I honestly have no idea if my compile times are quicker.
You can get benches from the horses mouth.
https://devblogs.microsoft.com/visualstudio/devdrive/
But the primary improvement comes from async A/V, not the file system.
ReFS offers other improvements such as CoW and instant file initialization which can benefit developers. From this standpoint, it is the correct choice over NTFS.
https://devblogs.microsoft.com/engineering-at-microsoft/dev-...