I develop on Windows 10 because I’m a .NET developer and really like the idea of WSL. But the more I use it the more frustrated I become by its file access performance. I started off using it for Git but now I just use Git bash in Poweshell (which also annoys me with its slowness).
I haven’t developed on an actual *NIX machine in years but recently I deployed something to my DO VPS and it “built” (php composer) in what felt like 1/100 of the time it was taking on my computer, whether running the Windows binaries in Powershell/CMD or the Linux binaries in WSL. Although I will say WSL is slower.
In fact, it was so fast that I’m about to call Microcenter and see about picking up a Thinkpad to install Linux on.
A major performance improvement is adding the working folders for my coding projects to the exclusion list of whatever antivirus solution is running. On low-specced machines I disable AV entirely, because I feel these days it is mostly snake oil anyway with zero days being commonplace.
To give another data point, for the software I'm developing (https://ossia.io/, a few hundred thousand LOC of modern C++ / Qt), a full build, with the same compiler version (clang-7), on the same machine, on the same pci-express NVMe disk takes 2 to 3 times slower on windows than on linux.
every time I have to build on windows I feel sad for all the developers forced to use this wretched environment.
For C++, on Windows I use MSVC compiler, and I'm usually happy with its performance. If your clang supports precompiled headers, try enabling them. This saves CPU time parsing standard headers, but also saves lots of IO accessing these random small include files, replacing the IO with large sequential read from the .pch.
I have a similar experience. I'm a developer on a product that has both C and C# components. The C component runs on both Windows and Linux and is by far the larger of the two while the C# GUI component is Windows-only.
Our main workstations are Windows but we also have a shared Linux development VM on our group's VM host. The VM host is not very powerful, it's a hand-me-down machine from another group that's around 7 years old at this point, with a low-clocked Bulldozer-based AMD Opteron CPU and spinning rust storage. In comparison my workstation has a 4th-gen Core i7 processor and a SATA SSD.
Despite the fact that my workstation should be around 2-3x as fast per thread and have massively better disk I/O the VM host still builds our C code in less than half the time. If I didn't have to maintain the C# code and use MS Office I'd switch to Linux as my main workstation OS in a heartbeat. (Note that the compilation process on neither platform uses parallel compilation so it's not that our VM is farming the compile out to more cores.)
In a similar situation - I've had reasonable success in Linux developing a .net framework c# app in VS Code with Mono from the Ubuntu PPA. Almost everything works but for the odd project calling native Windows DLLs, so I keep windows around for manipulating VS projects and compiling these oddities. Most dev can happen in Linux though.
So you have you core business logic and algorithms written in C and only the front end/GUI uses C#? Can you cleanly separate the two (via MVVC or a variation thereof)?
Does the whole application runs on C# and you have RPCs between the two? What does that mean for performance?
Cheers and thanks for any Insights about your setup.
Do you take the buffercache into account? If the server has more RAM unused by processes it probably uses it as a cache, which can be 3x quicker than a SSD.
Windows Defender is an abomination. It makes file operations on regular Windows also extremely slow. For example unpacking the Go 1.11 zip which is 8700 files takes a second with my PCIe SSD with Windows Defender disabled. Enable it and the extraction time rises to several minutes.
Can I plug System76 as an alternative? I feel like it's important to purchase Linux-native hardware. Microsoft has largely prevented competition in this market, and I think more viable options would benefit consumers. Also, S76 has pretty good hardware.
Thanks for the heads up (S76) but Linux runs on pretty much anything these days. This laptop is a Dell Inspiron with a 17" touch screen and lots of sensors. I'm running Arch Linux on it and everything is supported out of the box. The only tweak I have made towards hardware is changing the driver in use for the Synaptics mouse, which dmesg mentioned and will probably become the norm soon anyway.
There really is no such thing as MS native only anymore. I got Linux on here without accepting any obnoxious licenses and my laptop's price was partially subsidised by all the crap that I never even saw. To be honest, I'm not sure what the exact price breakdown really is on this thing but I do know that MS did not get in my way.
Dell are pretty Linux friendly, for example to update firmware I copy the new image to my /boot partition (EFI) and then use the built in option at boot to update the firmware - simples! No more farting around with turning swap into FAT32 for a while and a FreeDOS boot disc.
I'll plug the Librem 13/15. I have the 13 and I'm very happy with it (initially it had a bug in the firmware so my NVMe SSD that I bought separately wasn't bootable, but they fixed it very quickly).
Reasonable. .NET Core is obviously fine, but even .NET Framework stuff is largely runnable with up to date Mono, as MS is slowly pushing lots of previously 'system' libraries into NuGet. WPF is the only notable big dealbreaker.
> I started off using it for Git but now I just use Git bash in Poweshell (which also annoys me with its slowness).
This may not be a WSL issue. You don't want to mix Windows git (what you're calling git-bash) with Linux git (or MSYS2 git). Even a git status will cause them to trample on each other in ways I don't yet fully understand, and that will also slow them down very significantly as one tries to process a repo the other one has previously accessed. Pick one and stick with it for any given repo.
You may actually see better performance via docker or vm for linux under windows. Also, if you're using or can migrate to .Net core it works pretty well there.
I'm using containers for local services I'm not actively working on, even though the application is being deployed to windows, because it's been easier for me. I'd actually prefer Linux host at work, but there's legacy crap I need.
I do work remotely sometimes on my home hackintosh though.
I remember this issue (I commented in it a few years ago).
On the bright side, WSL as a development environment is no longer slow when it comes to real world usage. I've been using it for full time web app development for the last year and even made a video about my entire set up and configuration a week ago[0].
For example with WSL you can easily get <= 100ms code reloads even through a Docker for Windows volume on really big applications with thousands of files, such as a Rails app.
Even compiling ~200kb of SCSS through a few Webpack loaders (inside of Docker with a volume) takes ~600ms with Webpack's watcher. It's glorious.
I haven't had a single performance issue with a bunch of different Rails, Flask, Node, Phoenix and Jekyll apps (with and without Webpack). This is with all source code being mounted in from a spinning disk drive too, so it can get a lot faster.
So while maybe in a micro-benchmark, the WSL file system might be an order of magnitude slower (or worse) than native Linux, it doesn't really seem to matter much when you use it in practice.
> with WSL you can easily get <= 100ms code reloads even through a Docker for Windows volume
(Edited after watching your video.)
In your video it looks like you're running things in Docker containers. Even if you start containers using WSL, they still run in a separate Hyper-V virtual machine with a true Linux kernel, whereas WSL shares the Windows kernel and works by mapping Linux system calls directly to Windows kernel calls. When you run the "docker" command in WSL, it's just communicating with the Docker daemon running outside of WSL.
Docker runs this way on Windows because WSL does not implement all the Linux kernel system calls, only the most important ones needed by most applications, and the missing ones include some needed to run the Docker daemon.
All in all, this means that what you're talking about is not affected by the linked issue because it uses a different mechanism to access files (the Hyper-V driver rather than the WSL system call mapping). Although, if anything, I would expect Hyper-V to be even slower.
(Your edit makes my reply make a lot less since since you removed all of your original questions, but I'll leave my original reply, read the part after number 7)
My set up is basically this:
1. I use WSL as my day to day programming environment with the Ubuntu WSL terminal + tmux[0]. It's where I run a bunch of Linux tools and interact with my source code.
2. I have Docker for Windows installed (since the Docker daemon doesn't run directly in WSL yet due to missing Linux kernel features like iptables, etc.).
3. I installed Docker and Docker Compose inside of WSL but the daemon doesn't run in WSL. I just use the Docker CLI to communicate with Docker for Windows using DOCKER_HOST, so docker and docker-compose commands seamlessly work inside of WSL from my point of view[1].
4. All of my source code lives on a spinning disk drive outside of WSL which I edit with VSCode which is installed on Windows.
5. That source code drive is mounted into WSL using /etc/wsl.conf at / (but fstab works just as well)[2].
6. That source code drive is also shared with Docker for Windows and available to be used as a volume in any container.
7. All of my Dockerized web apps are running in Linux containers, but using this set up should be no problem if you use Windows containers I guess? I never used Windows containers, but that seems out of scope for WSL / Docker CLI. That comes down to Docker for Windows.
But, it's worth mentioning I have installed Ruby directly in WSL and didn't use Docker, and things are still just as fast as with Docker volumes. In fact, I run Jekyll directly in WSL without Docker because I really like live reload and I couldn't get that to work through Docker. My blog has like 200+ posts and 50+ drafts, and if I write a new blog post, Jekyll reloads the changes in about 3 seconds, and that's with Jekyll-Assets too. I have a feeling it wouldn't be that much faster even on native Linux since Jekyll is kind of slow, but I'm ok with a 3 second turn around considering it does so much good stuff for me.
Are you talking about running it from the command line?
I use Git almost every day and have it installed directly inside of WSL.
It's not slow for any projects I've worked on, but it's worth mentioning I'm not dealing with code bases with millions of lines of code and many tens of thousands of files.
Most of the code bases I work on have ~100kb LOC or less (most are much less), and hundreds or low thousands of files.
Grepping through 50,000+ files through a WSL mount on a non-SSD is pretty slow, but I haven't tried doing that on a native Linux system in a long time so I can't really say if that slowness is mostly WSL or grepping through a lot of files in general.
It rather depends on what "in practice" means in practice, though. I occasionally look at it with a test dominated by building small C programs and WSL remains several times slower than a Linux/ext4 install on the same hardware
I spent many years optimizing "stat-like" APIs for Picasa - Windows just feels very different than Linux once you're benchmarking.
It turns out Windows/SMB is very good at "give me all metadata over the wire for a directory" and not so fast at single file stat performance. On a high-latency network (e.g. Wi-Fi) the Windows approach is faster, but on a local disk (e.g., compiling code), Linux stat is faster.
This is off-topic, but is there any chance of bringing the Picasa desktop client back to the masses?
There's nothing out there that matches Picasa in speed for managing large collections (especially on Windows). The Picasa Image Viewer is lightning-fast, and I still use them both daily.
There are, however, some things that could be improved (besides the deprecated functionality that was gone when Picasa Online was taken away); e.g. "Export to Folder" takes its sweet time. But with no source out there, and no support from the developers, this will not, sadly, happen.
I'm mostly clueless about Windows, so bear with me, but that makes no sense to me.
If SMB has some "give me stat info for all stuff in a directory" API call then that's obviously faster over the network since it eliminates N roundtrips, but I'd still expect a Linux SMB host to beat a Windows SMB host at that since FS operations are faster, the Linux host would also understand that protocol.
Unless what you mean is that Windows has some kernel-level "stat N" interface, so it beats Linux by avoiding the syscall overhead, or having a FS that's more optimized for that use-case. But then that would also be faster when using a SMB mount on localhost, and whether it's over a high-latency network wouldn't matter (actually that would amortize some of the benefits).
I think the idea is that you're accessing files sparsely and/or randomly.
With the linux approach you avoid translating (from disk representation to syscall representation) metadata you don't need, and the in-memory disk cache saves having to re-read it (and some filesystems require a seek for each directory entry to read the inode data structure, which can also be avoided it you don't care about that particular stat).
With the windows approach, the kernel knows you want multiple files from the same directory, so it can send a (slightly more expensive) bulk stat request, using only one round trip[0]. On linux, the kernel doesn't know whether you're grabbing a.txt,b.txt,... (single directory-wide stat) or foo/.get,bar/.git,... (multiple single stats that could be pipelined) or just a single file, so it makes sense to use the cheapest request initially. If it then sees another stat in the same directory, it might make a bulk request, but that still incurred a extra round trip, and may have added useless processing overhead if you only needed two files.
TLDR: Access to distant memory is faster if assuptions can be made about your access patterns, access to local memory is faster if you access less of the local memory.
0: I'm aware of protocol-induced round trips, but I don't think it effects the reasoning.
> If SMB has some "give me stat info for all stuff in a directory" API call
It does, it supports FindFirstFile/FindNextFile[1], which returns a struct of name, attributes, size and timestamps per directory entry.
Now I'm not sure how Linux does things, but for NTFS, the data from FindFirstFile is pulled from the cached directory metadata, while the handle-based stat-like APIs operate on the file metadata. When the file is opened[2], the directory metadata is updated from the file metadata.
So while it does not have a "stat N" interface per se, the fact that it returns cached metadata in an explicit enumeration-style API should make it quite efficient.
If by "host" you mean the client rather than the server, and if I understand correctly, the problem I anticipate would be that the API doesn't allow you to use that cached metadata, even if the client has already received it, because there's no guarantee that when you query a file inside some folder, it'll be the same as it was when you enumerated that folder, so I'd assume you can't eliminate the round trip without changing the API. Not sure if I've understood the scenario correctly but that seems to be the issue to me.
Anectodally[1] javac dos full builds because reading everything is faster than statting every file comparing their compiled version. Eclipse works around this keeping a change list in memory, which had its own drawback with external changes pushing the workspace out of sync
[1] I can't find a source on this but I remember having read it a long time ago, so I'll leave it at that unless I can find an actual autoritative source.
You know any alternative to Picasa? Especially in regards to face recognition? Google Photos is objectively shit, as you need to upload all photos for that.
There is digikam for Linux (KDE) with facial recognition, I just started playing with it last night, I tested it on a small group of photos, and good so far,
I cant quite recall the exact number - but wasnt the packet count for an initial listing on windows SMB something like ~45 packets/steps in the transaction for each file?
Like I said - it was years ago, but I recall it being super chatty...
This is interesting, and to some extent, expected. I'd expect that emulating one OS on top of another is going to have performance challenges and constraints in general.
But the deep dive into aspects of Windows and Linux I/O reminded me that I'd love to see a new generation of clean sheet operating systems. Not another Unix-like OS. Not Windows. Something actually new and awesome. It's been a long time since Unix/BSD, Linux, and Windows NT/XP were introduced.
A few years ago, Microsoft developed a new operating system called Midori, but sadly they killed it. MS, Apple, and Google are each sitting on Carl Sagan money – billions and billions in cash. Would it hurt MS to spend a billion to build an awesome, clean-sheet OS? They could still support and update Windows 10 for as many years as they deemed optimal for their business, while also offering the new OS.
Would it cost more than a billion dollars to assemble an elite team of a few hundred people to build a new OS? Would Apple, Microsoft, or Google notice the billion dollar cost? Or even two billion?
If you think there's no point in a new OS, oh I think there's a lot of room for improvement right now. For one thing, we really ought to have Instant Computing in 2018. Just about everything we do on a computer should be instant, like 200 ms, maybe 100 ms. Opening an application and having it be fully ready for further action should be instant. Everything we do in an application should be instant, except for things like video transcoding or compressing a 1 GiB folder. OSes today are way, way too slow, which comes back to the file access performance issue on Windows. Even a Mac using a PCIe SSD fails to be instant for all sorts of tasks. They're all too damn slow.
We also need a fundamentally new security model. There's no way data should be leaving a user's computer as freely and opaquely as is the case now with all consumer OSes. Users should have much more control and insight into data leaving their machine. A good OS should also differentiate actual human users from software – the user model on nix is inadequate. Computers should be able to know when an action or authorization was physically committed by a human. That's just basic, and we don't have it. And I'm just scratching the surface of how much better security could be on general purpose OSes. There's so much more we could do.
Google is doing this, very publicly, with Fuchsia. Brand new kernel, not even POSIX compliant.
Microsoft is also doing this, in a different and substantially more expensive way [1]. Over the past several years they've been rewriting and unifying their disparate operating systems (Windows, Phone (before the fall), Xbox, etc) into a single modular kernel they're calling OneCore. Its more than likely that this work is based off of, if not totally contains, much of the NT kernel, but its the same line of thinking.
There is one massive rule when it comes to engineering management we see repeated over and over, yet no one listens: Do Not Rewrite. Period.
Apple is exemplary in this. We don't know how many changes they've made to iOS since its fork from MacOS long ago, which was based on BSD even longer ago. But have you used an iPad in recent history? Instant app starts. No lag. No-stutter rendering at 120fps. When HFS started giving them issues, they swapped it out with APFS. Apps are sandboxed completely from one-another, and have no way to break their sandbox even if the user wants them to. Etc. Comparing an iPad Pro's performance to most brand new Windows laptops is like fighting a low-orbit laser cannon with a civil war era musket. They've succeeded, however they managed to do that.
Point being, you don't rewrite. You learn, you adapt, and you iterate. We'll get there.
(And if you've read all this and then wondered "but isn't Fucshia a rewrite" you'd be right, and we should all have serious concerns about that OS ever seeing the light of day on a real product, and its quality once that happens. It won't be good. They can't even release a passable ChromeOS device [2])
> Would it cost more than a billion dollars to assemble an elite team of a few hundred people to build a new OS? Would Apple, Microsoft, or Google notice the billion dollar cost? Or even two billion?
It's not a matter of money, resources, or talent. For reference: The Mythical Man-Month, waterfall process, Windows Vista.
Building something of this scale and complexity from scratch will take a lot longer than even the most conservative estimate and will take many more years to shake out the bugs. Again, remember Windows Vista? And that was not nearly as revolutionary as the things you suggest.
Consider also that basically every single ground up "we are rethinking everything, and doing it right this time" OS rebuild from scratch has been a failure. There have been dozens, if not hundreds of examples.
I think the biggest problem is building the application ecosystem. iOS and Android were able to attract developers because they were the first movers on the computer phone platform. Convincing people to use a new OS for desktop (or any existing) computing without any apps would be difficult, and vice-versa convincing devs to support a platform with no users is also tough.
I think it's actually much easier now to get people on a new OS then it was a decade ago, or in the 90s, or the 80s. The web is so centric that a new OS with a great browser has won half the battle (I could write five pages on how massively better a browser could be than Chrome, Edge, Firefox, Safari, Opera, etc. and I love Firefox and Opera and Vivaldi and Brave.)
ChromeOS has partly proved the point, especially among college students and younger. Anyway, a serious effort at a new OS, a whole new stack, along with a new office suite and other applications, could succeed with the right (large) team and funding. People aren't very loyal to Microsoft – they're not loved. And Apple users have no idea how much better computers could be, but they would if someone showed them.
Would it cost more than a billion dollars to assemble an elite team of a few hundred people to build a new OS? Would Apple, Microsoft, or Google notice the billion dollar cost? Or even two billion?
This gives some idea of what is required to keep Linux moving forward. It would be nice if we could see similar stats for MS and Apple.
Yes, I like them a lot. A formally verified OS should be the standard these days (and formally verified compilers). But for that to be so, we'll need new programming languages (not Rust) and toolchains I guess.
> The NT file system API is designed around handles, not paths. Almost any operation requires opening the file first, which can be expensive. Even things that on the Win32 level seem to be a single call (e.g. DeleteFile) actually open and close the file under the hood. One of our biggest performance optimizations for DrvFs which we did several releases ago was the introduction of a new API that allows us to query file information without having to open it first.
Ouch that sounds painful... Is this why deleting gigs worth of files takes a bit? I could of sworn it's not a huge difference on Linux, at least when using the GUI, maybe when doing straight rm it's quicker.
Reminds me of a performance optimization I did at work. A previous developer had implemented a routine to clean up old files. The routine would enumerate all the files in a directory, and then ask for the file age, and if old enough delete the file.
The problem was that asking the file age given a path caused an open/close, as GetFileSizeEx expects a handle.
Now, at least on Windows, enumerating a directory gets[1] you not just the filename, but a record containing filename, attributes, size, creation and access timestamps. So all I had to do was simply merge the directory enumeration and age checking.
The result was several orders of magnitudes faster, especially if the directory was on a network share.
POSIX stuff is terrible for this sort of thing, but luckily it's often a straightforward fix once you know to look for it. Here's a similar fix for the old Win32 version of the silver searcher: https://github.com/kjk/the_silver_searcher/pull/7
Looks like the active Windows port (https://github.com/k-takata/the_silver_searcher-win32) doesn't have this change, but it's possible Mingw32 or whatever exposes this flag itself now (as it probably should have done originally).
Path-based I/O seems quite dangerous to me. If everything was path-based, you'd easily have inherent race conditions. You want to delete a directory? You stat() all the files, they look empty, so you delete them... but in between, another process writes to some file (or maybe the user forgets the file is being deleted and saves to something there), and suddenly you've deleted data you didn't expect. When you do things in a handle-based fashion, you know you're always referring to the same file (and can lock it to prevent updates, etc.), even if files are being moved around.
However, to answer your question of why removing a directory is slow... if you mean it's slow inside Explorer, a lot of it is shell-level processing (shell hooks etc.), not the file I/O itself. Especially if they're slow hooks -- e.g. if you have TortoiseGit with its cache enabled, for example, it can easily slow down deletions by a factor of 100x. But regarding the file I/O part, if it's really that slow at all, I think it's partly because the user-level API is path-based (because that's what people find easier to work with), whereas the system calls are mostly handle-based (because that's the more robust thing, as explained above... though it can also be faster, since a lot of work is already done for the handle and doesn't need to be re-performed on every access), so merely traversing the directory a/b/c requires opening and closing a, then opening and closing a/b, then opening and closing a/b/c, but even opening a/b/c requires internally processing a and b again, since they may no longer be the same things as before... this is O(n^2) in the directory depth. If you reduce it to O(n) by using the system calls and providing parent directory handles directly (NtOpenFile() with OBJECT_ATTRIBUTES->RootDirectory) then I think it should be faster and more robust.
You stat() all the files, they look empty, so you delete them... but in between, another process writes to some file (or maybe the user forgets the file is being deleted and saves to something there), and suddenly you've deleted data you didn't expect
This is fundamentally not any different between the systems, race conditions can happen either way. The user could write data to file right before the deletion recurses to the same directory and the handle-based deletion happens. Similarly the newly written data would be wiped out unintentionally.
For files for which access from different processes must be controlled explicitly there is locking. No filesystem or VFS is going to protect you from accidentally deleting stuff you're still using in another context.
> I could of sworn it's not a huge difference on Linux, at least when using the GUI, maybe when doing straight rm it's quicker.
Using Linux file management GUIs can be a disaster. Some years ago, I was massaging many GiB of text, using a crude hack, with scripts that did grep, sed, awk, etc. And I had many working directories with >10^4 tiny files each (maybe 10^5 even).
In terminal, everything was fine. But opening a GUI would swap out so much that the box would freeze. I don't know why. I just learned to avoid doing it.
Heh, that's fine, I don't mind the terminal or the GUI, as long as you're rationally using either and aren't blaming either for your problems. I see a lot of "I only use the terminal" type of people who screw up GUIs and the reverse can be said, GUI users who screw up the terminal, it's all about RTFM'ing sometimes, or just reading what's right in front of you.
Not only it's extremely surprising that there's no getting metadata without opening each file―and since DrvFs is only the WSL layer, apparently the system itself still to this day doesn't have such a feature.
But I'm now additionally baffled by how Total Commander managed to feel hella snappy ten years ago while navigating lots of dirs, whereas on MacOS all double-panel managers are meh, not least due to the rather slow navigation.
10 years ago there was less file system integration, user land virus scanning, kernel level virus scanning, os-hooks, OS-compatibility re-direction, and 32/64bit compatibility checks.
This was mostly added during NT6.0 era, which occured ~12 years ago. VISTA was the first OS using NT6.0 and VISTA was VERY much not in vogue ~12 years ago. In fact it was avoided like a plague as of 2008 (unless you were using 64bit, and had >=4GiB of RAM)
So many were using Windows-XP 32bit, or the NT5.2 kernel. Even those with >=4GiB of RAM were on Windows-XP 64bit, as VISTA had a ton of driver problems.
Thanks for sharing, that was an interesting read. I know the IO performance has always been the main gripe with WSL.
It makes me think more broadly about some of the trade offs with the discussions that happen on Github Issues.
It’s great that insights like this get pulled out in the discussions, and this could serve as excellent documentation. However, the discoverability will be difficult and it always takes some weeding through the low quality comments to piece the insights from contributors together.
I wonder how Github (and Microsoft) could better handle this. They could allow for flagging in-depth comments like this into a pinned section and they could be updated collaboratively in the future (Wiki?).
It also feels like a reputation system could help to motivate healthier discussions and could bury low quality “me too” comments and gripes. It’s not so bad in the linked example but I often encounter many rude comments aimed against volunteer OSS developers.
This particular post is pretty famous if you've been following the development of WSL. It's constantly linked to and referenced from other GitHub issues regarding WSL performance. So I think GitHub's current system is succeeding in that regard, although there are so many good points raised here that I wish it could get turned into a standalone blog post.
I can’t see MFT contention mentioned once. That’s what absolutely and totally destroys small file write performance. This affects source operations, WSL, compilers, file storage, everything.
And that’s so architecturally tied to the guts of NT you can’t fix it without pushing a new NTFS revision out. Which is risky and expensive.
Which is incidentally why no one at Microsoft even seems to mention it I suspect and just chips at trivial issues around the edges.
Bad show. NTFS is fundamentally broken. Go fix it.
Edit: my experience comes from nearly two decades of trying to squeeze the last bit of juice out of windows unsuccessfully. My conclusion is don’t bother. Ext4, ZFS and APFS are at least an order of magnitude more productive and this is a measurable gain.
Perhaps we didn't read the same article. What it says that the root of problem is the Windows IO subsystem architecture. Change NTFS for anything and you will get the same problem.
But that’s not the case. The root cause is the MFT and NTFS architecture. People fail to mention that because the problem is harder to fix. It’s almost that there is a “do not speak bad of NTFS” going on.
You can demonstrate this by using a third party file system driver for NT. when NTFS on its own is eliminated the performance is much much better. This is a neat little differential analysis which is conclusive. I can’t remember the product I used when I evaluated this about 8 years ago unfortunately.
I think this is a very good example of how windows is different in its goals and designs from Linux. I have a feeling this is the reason Linux has had a hard time catching on, on the desktop. Its easy to complain about a slow filesystem, but Microsoft lives in a different world, where other priorities exist. For someone building a server running a database serving loads of people Linux is a no brainer. You can pick the parts you want, shrink wrap it and fire it up without any fat. On a desktop machine, you want to be able to update drivers in the background without rebooting, you want virus scanners, and you want to have driver ready the moment the user plug's in a new device. Both Windows and Linux is for the most part very well engineered, but with very different priorities.
I’m very confused by your post. You start off talking about desktop machines but NT was actually engineered for servers and then later ported to the desktop. You then describe a bunch of features Linux does better than Windows (eg updating drivers without a reboot).
I think a more reasonable argument is to make is just that Windows is engineered differently to Linux. There’s definitely advantages and disadvantages to each approach but ultimately it’s a question of personal preference.
NT is engineered for a different category of servers, though - it's a workgroup server first (originally its chief competitor was NetWare), and a Web/Internet server second. That drives a different set of priorities.
For example, as someone elsewhere in the comments pointed out, NT does file access in a way that works very well when accessing network shares. That's a pretty core use case for Windows on business workstations, where it's common for people to store all the most important files they work with on a network share, for easier collaboration with other team members.
NT was architected to handle high-end workstations from day one — there’s a reason why running the GUI was mandatory even when the resource costs were fairly substantial.
Check out e.g. https://en.wikipedia.org/wiki/Windows_NT_3.1 for the history of that era. The big selling point was that your business could code against one API everywhere, rather than having DOS PCs and expensive Unix, VAX, etc. hardware which was completely different and only a few people on staff were comfortable with.
Drivers in Linux live in the kernel. Whenever the kernel is updated a reboot is required (in most distros). Hence your assertion that Linux updates drivers without a reboot better than windows does is questionable.
> On a desktop machine, you want to be able to update drivers in the background without rebooting, you want virus scanners, and you want to have driver ready the moment the user plug's in a new device.
With the exception of the virus scanner these actually sound like arguments in favour of Linux, in my experience.
(Although there are also excellent virus scanners available for Linux anyway)
I’m pretty confused by this post. What would you identify their priorities as?
Regardless, as a person not a fan of windows (not worth learning a unix alternative), I would argue it’s the polish that makes the experience worth it, not some better engineered experience. For instance: working with variable DPI seems to be trivial, whereas it still seems years off in Linux. Same with printers and notifications and internet and almost everything. These aren’t feats of engineering per se, but they do indicate forethought I deeply appreciate when I do use windows.
I would hesitate to ascribe too much purpose to everything you see. Microsoft is a huge company with conflicting political factions and a deep ethos of maintaining backwards compatibility so there are plenty of things which people didn’t anticipate ending up where they are but which are risky to change.
One big factor is the lost era under Ballmer. Stack ranking meant that the top n% of workers got bonuses and the bottom n% were fired, and management reportedly heavily favored new features over maintenance. Since the future was WinFS and touching something core like NTFS would be a compatibility risk, you really wouldn’t have an incentive to make a change without a lot of customer demand.
As a c# Dev, I am constantly annoyed that Windows updates and sometimes installs require reboots or stopping all user activity, while I've never had to reboot or block during an upgrade on ubuntu
To be fair, a lot of Linux updates require a reboot or at least a logout to properly take effect, too. Windows is just very aggressive about forcing you to upgrade and reboot, which does undeniably have security benefits when you consider that Windows has a lot of non-technical users and a huge attack surface. At least they have relaxed it a bit, the frequent forced reboots caused me some serious problems on a Windows machine I had controlling a CNC machine.
Most of these things are coincidental byproducts of how Windows (NT) is designed, not carefully envisioned trade offs that are what make Windows Ready for the Desktop (tm).
For some counterexamples of how those designs make things harder and more irritating, look at file locking and how essentially every Windows update forces a reboot, that is pretty damn user unfriendly.
Even without file locking, how would live updates work when processes communicate with each other and potentially share files/libraries? I feel like file locking isn't really the core problem here.
They mention that file performance decreases with the numbers of filters that are attached to the NTFS drive. Is there a way to list the filters to determine which ones (and how many) are applied on your system?
Interesting this article never said "poorly architected". The conclusion that the issue is peanut buttered points at that. Instead of looking into the system for things to optimize is there any proposal or initiative to rework it at a higher level?
> Noting, as an aside, that it isn't really all that necessary for MSFT to work on the project, because I gather there are at least 684 FOSS NT kernel developers available, and qualified, and willing, to volunteer their time to work on projects like that. I assume that's why all those people upvoted, anyway. With a team that size, stabilizing WinBtrfs will take no time at all.
On similar lines, one of the usual gripe with windows is the horrible file copy estimates, Raymond Chen wrote an article on it to explain why it is[1].
The Old New Thing is a fantastic blog especially if you develop in Windows. I used to read it religiously until 5 years back and stopped reading about time when Google Reader was retired and somehow didn't setup in my new flow of reading blogs. Thanks for reminding me about this blog and article again.
You're being downvoted, but it's true about linux for at least two reasons to this day:
1. dm-crypt threads are unfair to the rest of the system's processes [1]. On dmcrypt systems, regular user processes can effectively raise their process scheduling priority in a multithreaded fashion by generating heavy IO on dmcrypt storage.
2. Under memory pressure, the VM system in linux will enter a thrashing state even when there is no swap configured at all. I don't have a reference on hand, but it's been discussed on lkml multiple times without solution. I suspect the recent PSI changes are intended as a step towards a solution though. What happens is clean, file-backed pages for things like shared libraries and executable programs become a thrashing set resembling anonymous memory swapping under memory pressure. As various processes get scheduled, they access pages which were recently discarded from the page cache as evictable due to their clean file-backed status when other processes ran under pressure, and now must be read back in from the backing store. This ping-ponging continues dragging everything down until either an OOM occurs or pressure is otherwise relieved. This often manifests as a pausing/freezing desktop with the disk activity light blazing, and it's only made worse by the aforementioned dmcrypt problem if these files reside on such volumes.
This behaviour - and perhaps other causes with similar issues for desktop users, is what drove me away from helping to make Linux ‘ready for the desktop’. The kernel philosophy was incompatible with the needs of desktop users. Those developing for the desktop (like myself) didn’t have the required expertise to make the kernel do the Right a Thing, and couldn’t find enough people willing or able to help.
Things may have changed over the years - I’ve been running a Linux desktop recently and haven’t seen this kind of issue yet (the kind where you need to use magic keys to ask the kernel to kill all, sync and reboot) but reading your post, perhaps this is because RAM is much more plentiful these days.
I’ve attempted to use Linux and desktop freezing is the norm even on machines that run fine under windows. Admittedly the machines might be underpowered but that does call into question the commonly held belief that desktop Linux is better for low spec machines.
I have had issues at times in the past but with things like core dumps on systems with spinning disk and 128gb ram. The OOM on linux can be brutally frustrating. But it's still light years ahead of windows for development...
> At least windows doesn't freeze your whole desktop under heavy IO.
It does it as well on Windows and unlike Linux, Windows still has a risk of permanent damage under low space available, it was even worse in the XP days but these issues are still there.
I've experienced a total freeze before once one of my programs started swapping/thrashing. The entire desktop froze, not just the one program. This was in the past two years or so, so it's not a solved problem.
Case in point: I recently tried unzipping the Boost library on an up-to-date Windows 10, and after trying to move the frozen progress window after a minute, the whole desktop promptly crashed. I have to say, the experience is better than it used to be, because at least the taskbar reappeared on its own. (Decompression succeeded on the second attempt after leaving the whole computer well alone ... but it certainly took its time even on a high-end desktop computer.)
Could you please leave personal swipes out of your comments here? They have a degrading effect on discussion and evoke worse from others. Your comment would be just fine without the second sentence.
The issue seems to be with how Windows handles file system operations. They allow filters to operate over each request. These filters are like hardware drivers in that they are created and maintained by third parties. So, MS has no real ability to change how things work because they don't have absolute control in the same way that Linux does over the operations (Linux device drivers are part of the kernel)
Not sure if I'm being naive, but couldn't they make it so that if there are no filters present on a given volume, then the entire filter stack is bypassed and the directory nodes are served from a cache managed by the file system at the root of the volume? That way developers who really want performance can make sure they have no filter drivers installed on that volume.
The linked post notes that, even on a default Windows install, you already have a sizable stack of filter drivers active. Not sure whether that only applies to the C: volume or to other ones as well.
If they really wanted to, they could mount an ext4 fs as a single blob on disk directly into WSL. That won't really help if you want to run a GUI editor in windows against that identical directory though.
NTOS (Windows NT's kernel) was originally designed to be backward compatible with Windows (DOS era), OS/2 and POSIX. Designing such a kernel was considered a performance at the time but probably cost its fair share of bloat that kept growing with the years. Also, it's not surprising that Linux is optimized for operating on files since the UNIX philosophy is about considering everything as a file, something that Dave Cutler (NTOS main designer) was known for criticizing[1].
I pray daily that more of my fellow-programmers may find the means of freeing themselves from the curse of compatibility. (Dijkstra)
As another data point here: I had to add a TKinter-based loading screen to a PyQt5 app I wrote for work because the app - which takes half a second to start up on Linux - takes nearly a minute on Windows the first time around (and multiple dozens of seconds on subsequent runs) and I know firsthand that my end-users are unlikely to wait that long unless they can see something loading. I suspect it has to do with the sheer number of file accesses for each of the Python libraries and underlying DLLs and such.
> A Win32 path like C:\dir\file gets translated to an NT path like \??\C:\dir\file, where \??\C: is a symlink in Object Manager to a device object like \Device\HarddiskVolume4
Cool! Can the "NT paths" be used directly on Windows? As far as I know the ordinary Win32/NTFS system doesn't even have symlinks and this feels quite a handicap.
Note that this is behind a "developper flag", i.e. it needs elevation if you have not checked "developper mode" in the W10 parameters. (I guess they don't want the cost of supporting ordinary people building circular dependencies)
It depends on which API the application uses to pass paths to the OS and whether its path handling library does internal parsing that is or is not aware of those prefixes.
For example rust has support[0] for the magical file path prefixes[1] that can be used to escape path length restrictions, reserved identifiers and some other limitations. This is used by default whenever you normalize a part, which means it has fewer restrictions than windows explorer has by default.
These require user-mode round trips for path traversal, so comparable to FUSE, I believe.
So it is relatively easy to add hooks of your own that get called as if they were file reads; NTFS calls these "Reparse Points". Linux would do this with named pipes.
I am just a user of these things, haven't dug into any implementation details recently, but I guess that the stacks to support these features in Linux and Windows each have their cache etc design decisions that have led to current state of filesystem performance under typical use-cases these days.
I'm not a super expert but I think that is the case and you even need them if you want to use some advanced features like paths that are longer than MAX_PATH or these virtual mount kinds of things.
But IMO it's ugly as hell and just begging for compatibility problems. So don't go there, at least not to get super long paths. 260 bytes should be enough for anybody.
Yes they can, though one problems is that because NT paths are less restrictive than Win32 paths, you can create files / directories that you then have trouble doing things with using standard tools
Notably the Windows shell does not support longlong file names, like, at all. The Windows shell and Win32 APIs are also easily confused by files using the NTFS POSIX namespace, which may have \ in their name.
I still scracth my head as to why doing a kernel update of Ubuntu running under Hyper-V on a spinning disk is so horrifically slow. If I migrate the VM while it's running to SSD, let it finish, then migrate it back to spinning, it's faster then just letting it update in place. This is my device desktop, otherwise I would put in on SSD full time.
The above comments make me think something like the stat calls may be the issue and moving the running vm to SSD hides the problem. It obviously isn't raw disk rate at the point.
Is the Ubuntu VM's / on a fixed or dynamic VHD? The former provides (modulo filesystem fragmentation) linear addressing of virtual LBAs to physical LBAs, and the latter can be heavily fragmented and result in sequential read/write operations becoming random access, which on a HDD kills performance.
My advice for anyone running VMs is if they're on HDDs or they're highly tuned for disk usage (databases for example), use fixed VHDs, otherwise use dynamic.
I still scracth my head as to why doing a kernel update of Ubuntu running under Hyper-V on a spinning disk is so horrifically slow.
Define slow please.
For a laugh I picked a random VM (VMWare) at work and ran (I did apt update first):
# time apt upgrade
...
82 to upgrade, 5 to newly install, 0 to remove and 0 not to upgrade
...
real 6m16.015s
user 2m38.936s
sys 0m55.216s
The updates included two client server DB engines (Clickhouse and Postgresql) the fileserving thingie (Samba) and a few other bits. The reboot takes about 90 seconds before the webby interface appears for rspamd.
Specifics,Host OS is Windows 10 Pro,virtual machine is Hyper-V, Guest OS is Ubuntu, Dynamic VHXD, Storage Space Pool with 4 HDD and 1 SSD, Windows Filesystem ReFS.
Only observe this specifically with kernel updates. Everything updates as I would expect on a HDD.
Slow is ~10-20 minutes. Which is why it is faster to migrate the running VM from the Storage pool, onto a single SSD, complete the ubuntu kernel update and migrate back.
Have you cleaned away the old kernel packages? Kernel package operations on Debian and Ubuntu are accidentally quadratic with a high time constant (there is a package script for each kernel that iterates over all installed kernels) so you want to make sure not to let them accumulate.
The build speed difference for Android (Gradle build system) between Windows and Linux is extremely noticeable especially for large projects, where you can see even almost 50% faster builds.
Keep in mind that Linux filesystems are all implemented in the kernel (FUSE aside), and WSL doesn't run a Linux kernel - it just emulates syscalls for userland to work. So there's no ext drivers in there, or any other standard Linux FS drivers.
tldr: discussion is about why disk access under WSL is slower than under Linux, mostly due to the different design constraints of Linux vs Windows for file systems.
An interesting comment from insider. We all knows from various benchmarks that Microsoft Windows' filesystem access performance was way too worse than Linux kernel not only WSL but also Win32 Subsystem too.
Also, process creating performance is worse on Windows. I wonder if it is also the case of "Death by thousand cuts"
Windows performance is abysmal. Instead of beating around the bush, they should just state that it is the combination of everything that makes Windows dog slow compared to either Mac or Linux. Someone said it on that thread, but Windows really should be a DE on Linux now instead of being its own OS.
I develop on Windows 10 because I’m a .NET developer and really like the idea of WSL. But the more I use it the more frustrated I become by its file access performance. I started off using it for Git but now I just use Git bash in Poweshell (which also annoys me with its slowness).
I haven’t developed on an actual *NIX machine in years but recently I deployed something to my DO VPS and it “built” (php composer) in what felt like 1/100 of the time it was taking on my computer, whether running the Windows binaries in Powershell/CMD or the Linux binaries in WSL. Although I will say WSL is slower.
In fact, it was so fast that I’m about to call Microcenter and see about picking up a Thinkpad to install Linux on.
A major performance improvement is adding the working folders for my coding projects to the exclusion list of whatever antivirus solution is running. On low-specced machines I disable AV entirely, because I feel these days it is mostly snake oil anyway with zero days being commonplace.
Uhh what? Software is probably more secure now than ever before. I don't think AV software is snake oil at all.
10 replies →
To give another data point, for the software I'm developing (https://ossia.io/, a few hundred thousand LOC of modern C++ / Qt), a full build, with the same compiler version (clang-7), on the same machine, on the same pci-express NVMe disk takes 2 to 3 times slower on windows than on linux.
every time I have to build on windows I feel sad for all the developers forced to use this wretched environment.
For C++, on Windows I use MSVC compiler, and I'm usually happy with its performance. If your clang supports precompiled headers, try enabling them. This saves CPU time parsing standard headers, but also saves lots of IO accessing these random small include files, replacing the IO with large sequential read from the .pch.
1 reply →
I have a similar experience. I'm a developer on a product that has both C and C# components. The C component runs on both Windows and Linux and is by far the larger of the two while the C# GUI component is Windows-only.
Our main workstations are Windows but we also have a shared Linux development VM on our group's VM host. The VM host is not very powerful, it's a hand-me-down machine from another group that's around 7 years old at this point, with a low-clocked Bulldozer-based AMD Opteron CPU and spinning rust storage. In comparison my workstation has a 4th-gen Core i7 processor and a SATA SSD.
Despite the fact that my workstation should be around 2-3x as fast per thread and have massively better disk I/O the VM host still builds our C code in less than half the time. If I didn't have to maintain the C# code and use MS Office I'd switch to Linux as my main workstation OS in a heartbeat. (Note that the compilation process on neither platform uses parallel compilation so it's not that our VM is farming the compile out to more cores.)
In a similar situation - I've had reasonable success in Linux developing a .net framework c# app in VS Code with Mono from the Ubuntu PPA. Almost everything works but for the odd project calling native Windows DLLs, so I keep windows around for manipulating VS projects and compiling these oddities. Most dev can happen in Linux though.
1 reply →
I am not a C# dev, so please bear with me.
So you have you core business logic and algorithms written in C and only the front end/GUI uses C#? Can you cleanly separate the two (via MVVC or a variation thereof)?
Does the whole application runs on C# and you have RPCs between the two? What does that mean for performance?
Cheers and thanks for any Insights about your setup.
1 reply →
Do you take the buffercache into account? If the server has more RAM unused by processes it probably uses it as a cache, which can be 3x quicker than a SSD.
1 reply →
Why not flip to linux and run Windows in a VM?
This helped me: https://github.com/Microsoft/WSL/issues/1932#issuecomment-40...
Microsoft made some change a year or so ago that brought more of WSL into Windows Defender's view.
Windows Defender is an abomination. It makes file operations on regular Windows also extremely slow. For example unpacking the Go 1.11 zip which is 8700 files takes a second with my PCIe SSD with Windows Defender disabled. Enable it and the extraction time rises to several minutes.
1 reply →
Can I plug System76 as an alternative? I feel like it's important to purchase Linux-native hardware. Microsoft has largely prevented competition in this market, and I think more viable options would benefit consumers. Also, S76 has pretty good hardware.
Thanks for the heads up (S76) but Linux runs on pretty much anything these days. This laptop is a Dell Inspiron with a 17" touch screen and lots of sensors. I'm running Arch Linux on it and everything is supported out of the box. The only tweak I have made towards hardware is changing the driver in use for the Synaptics mouse, which dmesg mentioned and will probably become the norm soon anyway.
There really is no such thing as MS native only anymore. I got Linux on here without accepting any obnoxious licenses and my laptop's price was partially subsidised by all the crap that I never even saw. To be honest, I'm not sure what the exact price breakdown really is on this thing but I do know that MS did not get in my way.
Dell are pretty Linux friendly, for example to update firmware I copy the new image to my /boot partition (EFI) and then use the built in option at boot to update the firmware - simples! No more farting around with turning swap into FAT32 for a while and a FreeDOS boot disc.
4 replies →
I'll plug the Librem 13/15. I have the 13 and I'm very happy with it (initially it had a bug in the firmware so my NVMe SSD that I bought separately wasn't bootable, but they fixed it very quickly).
+1 for S76 HW guts, but I find their laptop chassis to have been really flimsy in the past - where screws fell out and into the case of 3 laptops.
They were, in the past, using CLEO as an OEM....
They came out with that custom steel and wood case all made in the USA recently, but don't know if they are building their own laptop HW these days.
2 replies →
Sadly they are overpriced to hell.
Curious just how feasible doing . NET dev in Linux would be? Is it a non start? Is it inconvenient and rough? Is it fine if you give it enough love?
If I only messed with .NET Core I think it would be fine. I already use JetBrains Rider in place of VS.
The problem is that many systems still rely on .NET Framework which is a non start on Linux.
Also stuff like this:
https://github.com/dotnet/cli/issues/3093#issuecomment-44343...
Trust, is an issue.
1 reply →
Depends on if you're just writing pure libraries, or actually doing interesting things like UI or linking to 3rd party drivers.
I think the experience with mssql is better on linux now, but I imagine anything slightly outside the box db-wise may break you.
1 reply →
It's the lack of visual studio that is often the nonstarter.
12 replies →
Reasonable. .NET Core is obviously fine, but even .NET Framework stuff is largely runnable with up to date Mono, as MS is slowly pushing lots of previously 'system' libraries into NuGet. WPF is the only notable big dealbreaker.
Web stuff is fine.
> I started off using it for Git but now I just use Git bash in Poweshell (which also annoys me with its slowness).
This may not be a WSL issue. You don't want to mix Windows git (what you're calling git-bash) with Linux git (or MSYS2 git). Even a git status will cause them to trample on each other in ways I don't yet fully understand, and that will also slow them down very significantly as one tries to process a repo the other one has previously accessed. Pick one and stick with it for any given repo.
You may actually see better performance via docker or vm for linux under windows. Also, if you're using or can migrate to .Net core it works pretty well there.
I'm using containers for local services I'm not actively working on, even though the application is being deployed to windows, because it's been easier for me. I'd actually prefer Linux host at work, but there's legacy crap I need.
I do work remotely sometimes on my home hackintosh though.
Same. I changed to cygwin for my git workflow because WSL is just too slow.
If you had used WSLGit (https://github.com/andy-5/wslgit), you could use Cyg-Git (https://github.com/nukata/cyg-git) instead. It provides a "Git for Windows" virtually for Go and VSCode. You only need Cygwin and its packages, including Cygwin's git.
I remember this issue (I commented in it a few years ago).
On the bright side, WSL as a development environment is no longer slow when it comes to real world usage. I've been using it for full time web app development for the last year and even made a video about my entire set up and configuration a week ago[0].
For example with WSL you can easily get <= 100ms code reloads even through a Docker for Windows volume on really big applications with thousands of files, such as a Rails app.
Even compiling ~200kb of SCSS through a few Webpack loaders (inside of Docker with a volume) takes ~600ms with Webpack's watcher. It's glorious.
I haven't had a single performance issue with a bunch of different Rails, Flask, Node, Phoenix and Jekyll apps (with and without Webpack). This is with all source code being mounted in from a spinning disk drive too, so it can get a lot faster.
So while maybe in a micro-benchmark, the WSL file system might be an order of magnitude slower (or worse) than native Linux, it doesn't really seem to matter much when you use it in practice.
[0]: https://www.youtube.com/watch?v=5gu8wWX3Ob4
> with WSL you can easily get <= 100ms code reloads even through a Docker for Windows volume
(Edited after watching your video.)
In your video it looks like you're running things in Docker containers. Even if you start containers using WSL, they still run in a separate Hyper-V virtual machine with a true Linux kernel, whereas WSL shares the Windows kernel and works by mapping Linux system calls directly to Windows kernel calls. When you run the "docker" command in WSL, it's just communicating with the Docker daemon running outside of WSL.
Docker runs this way on Windows because WSL does not implement all the Linux kernel system calls, only the most important ones needed by most applications, and the missing ones include some needed to run the Docker daemon.
All in all, this means that what you're talking about is not affected by the linked issue because it uses a different mechanism to access files (the Hyper-V driver rather than the WSL system call mapping). Although, if anything, I would expect Hyper-V to be even slower.
(Your edit makes my reply make a lot less since since you removed all of your original questions, but I'll leave my original reply, read the part after number 7)
My set up is basically this:
1. I use WSL as my day to day programming environment with the Ubuntu WSL terminal + tmux[0]. It's where I run a bunch of Linux tools and interact with my source code.
2. I have Docker for Windows installed (since the Docker daemon doesn't run directly in WSL yet due to missing Linux kernel features like iptables, etc.).
3. I installed Docker and Docker Compose inside of WSL but the daemon doesn't run in WSL. I just use the Docker CLI to communicate with Docker for Windows using DOCKER_HOST, so docker and docker-compose commands seamlessly work inside of WSL from my point of view[1].
4. All of my source code lives on a spinning disk drive outside of WSL which I edit with VSCode which is installed on Windows.
5. That source code drive is mounted into WSL using /etc/wsl.conf at / (but fstab works just as well)[2].
6. That source code drive is also shared with Docker for Windows and available to be used as a volume in any container.
7. All of my Dockerized web apps are running in Linux containers, but using this set up should be no problem if you use Windows containers I guess? I never used Windows containers, but that seems out of scope for WSL / Docker CLI. That comes down to Docker for Windows.
But, it's worth mentioning I have installed Ruby directly in WSL and didn't use Docker, and things are still just as fast as with Docker volumes. In fact, I run Jekyll directly in WSL without Docker because I really like live reload and I couldn't get that to work through Docker. My blog has like 200+ posts and 50+ drafts, and if I write a new blog post, Jekyll reloads the changes in about 3 seconds, and that's with Jekyll-Assets too. I have a feeling it wouldn't be that much faster even on native Linux since Jekyll is kind of slow, but I'm ok with a 3 second turn around considering it does so much good stuff for me.
[0]: https://nickjanetakis.com/blog/conemu-vs-hyper-vs-terminus-v...
[1]: https://nickjanetakis.com/blog/setting-up-docker-for-windows...
[2]: https://github.com/nickjj/dotfiles/blob/master/etc/wsl.conf
1 reply →
Git runs really slow on it. It runs much faster when turning off Windows defender (but still slow).
Are you talking about running it from the command line?
I use Git almost every day and have it installed directly inside of WSL.
It's not slow for any projects I've worked on, but it's worth mentioning I'm not dealing with code bases with millions of lines of code and many tens of thousands of files.
Most of the code bases I work on have ~100kb LOC or less (most are much less), and hundreds or low thousands of files.
Grepping through 50,000+ files through a WSL mount on a non-SSD is pretty slow, but I haven't tried doing that on a native Linux system in a long time so I can't really say if that slowness is mostly WSL or grepping through a lot of files in general.
3 replies →
It rather depends on what "in practice" means in practice, though. I occasionally look at it with a test dominated by building small C programs and WSL remains several times slower than a Linux/ext4 install on the same hardware
But the point is that if your compilation time is less than eg 100ms on WSL, it doesn’t really matter if it’s 2 or 100 times slower than ext4.
4 replies →
I spent many years optimizing "stat-like" APIs for Picasa - Windows just feels very different than Linux once you're benchmarking.
It turns out Windows/SMB is very good at "give me all metadata over the wire for a directory" and not so fast at single file stat performance. On a high-latency network (e.g. Wi-Fi) the Windows approach is faster, but on a local disk (e.g., compiling code), Linux stat is faster.
You've done an amazing job.
This is off-topic, but is there any chance of bringing the Picasa desktop client back to the masses?
There's nothing out there that matches Picasa in speed for managing large collections (especially on Windows). The Picasa Image Viewer is lightning-fast, and I still use them both daily.
There are, however, some things that could be improved (besides the deprecated functionality that was gone when Picasa Online was taken away); e.g. "Export to Folder" takes its sweet time. But with no source out there, and no support from the developers, this will not, sadly, happen.
Have you looked at Adobe bridge? It's now free as long as you sign up for a (free) Adobe account.
I'm mostly clueless about Windows, so bear with me, but that makes no sense to me.
If SMB has some "give me stat info for all stuff in a directory" API call then that's obviously faster over the network since it eliminates N roundtrips, but I'd still expect a Linux SMB host to beat a Windows SMB host at that since FS operations are faster, the Linux host would also understand that protocol.
Unless what you mean is that Windows has some kernel-level "stat N" interface, so it beats Linux by avoiding the syscall overhead, or having a FS that's more optimized for that use-case. But then that would also be faster when using a SMB mount on localhost, and whether it's over a high-latency network wouldn't matter (actually that would amortize some of the benefits).
I think the idea is that you're accessing files sparsely and/or randomly.
With the linux approach you avoid translating (from disk representation to syscall representation) metadata you don't need, and the in-memory disk cache saves having to re-read it (and some filesystems require a seek for each directory entry to read the inode data structure, which can also be avoided it you don't care about that particular stat).
With the windows approach, the kernel knows you want multiple files from the same directory, so it can send a (slightly more expensive) bulk stat request, using only one round trip[0]. On linux, the kernel doesn't know whether you're grabbing a.txt,b.txt,... (single directory-wide stat) or foo/.get,bar/.git,... (multiple single stats that could be pipelined) or just a single file, so it makes sense to use the cheapest request initially. If it then sees another stat in the same directory, it might make a bulk request, but that still incurred a extra round trip, and may have added useless processing overhead if you only needed two files.
TLDR: Access to distant memory is faster if assuptions can be made about your access patterns, access to local memory is faster if you access less of the local memory.
0: I'm aware of protocol-induced round trips, but I don't think it effects the reasoning.
1 reply →
> If SMB has some "give me stat info for all stuff in a directory" API call
It does, it supports FindFirstFile/FindNextFile[1], which returns a struct of name, attributes, size and timestamps per directory entry.
Now I'm not sure how Linux does things, but for NTFS, the data from FindFirstFile is pulled from the cached directory metadata, while the handle-based stat-like APIs operate on the file metadata. When the file is opened[2], the directory metadata is updated from the file metadata.
So while it does not have a "stat N" interface per se, the fact that it returns cached metadata in an explicit enumeration-style API should make it quite efficient.
[1]: https://docs.microsoft.com/en-us/windows/desktop/api/fileapi... [2]: https://blogs.msdn.microsoft.com/oldnewthing/20111226-00/?p=...
2 replies →
If by "host" you mean the client rather than the server, and if I understand correctly, the problem I anticipate would be that the API doesn't allow you to use that cached metadata, even if the client has already received it, because there's no guarantee that when you query a file inside some folder, it'll be the same as it was when you enumerated that folder, so I'd assume you can't eliminate the round trip without changing the API. Not sure if I've understood the scenario correctly but that seems to be the issue to me.
Anectodally[1] javac dos full builds because reading everything is faster than statting every file comparing their compiled version. Eclipse works around this keeping a change list in memory, which had its own drawback with external changes pushing the workspace out of sync
[1] I can't find a source on this but I remember having read it a long time ago, so I'll leave it at that unless I can find an actual autoritative source.
Interesting, tx for sharing :)
You know any alternative to Picasa? Especially in regards to face recognition? Google Photos is objectively shit, as you need to upload all photos for that.
There is digikam for Linux (KDE) with facial recognition, I just started playing with it last night, I tested it on a small group of photos, and good so far,
1 reply →
Depending on your platform - Apple's Photos is pretty good with facial recognition, and it's all done on device.
1 reply →
I cant quite recall the exact number - but wasnt the packet count for an initial listing on windows SMB something like ~45 packets/steps in the transaction for each file?
Like I said - it was years ago, but I recall it being super chatty...
This is interesting, and to some extent, expected. I'd expect that emulating one OS on top of another is going to have performance challenges and constraints in general.
But the deep dive into aspects of Windows and Linux I/O reminded me that I'd love to see a new generation of clean sheet operating systems. Not another Unix-like OS. Not Windows. Something actually new and awesome. It's been a long time since Unix/BSD, Linux, and Windows NT/XP were introduced.
A few years ago, Microsoft developed a new operating system called Midori, but sadly they killed it. MS, Apple, and Google are each sitting on Carl Sagan money – billions and billions in cash. Would it hurt MS to spend a billion to build an awesome, clean-sheet OS? They could still support and update Windows 10 for as many years as they deemed optimal for their business, while also offering the new OS.
Would it cost more than a billion dollars to assemble an elite team of a few hundred people to build a new OS? Would Apple, Microsoft, or Google notice the billion dollar cost? Or even two billion?
If you think there's no point in a new OS, oh I think there's a lot of room for improvement right now. For one thing, we really ought to have Instant Computing in 2018. Just about everything we do on a computer should be instant, like 200 ms, maybe 100 ms. Opening an application and having it be fully ready for further action should be instant. Everything we do in an application should be instant, except for things like video transcoding or compressing a 1 GiB folder. OSes today are way, way too slow, which comes back to the file access performance issue on Windows. Even a Mac using a PCIe SSD fails to be instant for all sorts of tasks. They're all too damn slow.
We also need a fundamentally new security model. There's no way data should be leaving a user's computer as freely and opaquely as is the case now with all consumer OSes. Users should have much more control and insight into data leaving their machine. A good OS should also differentiate actual human users from software – the user model on nix is inadequate. Computers should be able to know when an action or authorization was physically committed by a human. That's just basic, and we don't have it. And I'm just scratching the surface of how much better security could be on general purpose OSes. There's so much more we could do.
Google is doing this, very publicly, with Fuchsia. Brand new kernel, not even POSIX compliant.
Microsoft is also doing this, in a different and substantially more expensive way [1]. Over the past several years they've been rewriting and unifying their disparate operating systems (Windows, Phone (before the fall), Xbox, etc) into a single modular kernel they're calling OneCore. Its more than likely that this work is based off of, if not totally contains, much of the NT kernel, but its the same line of thinking.
There is one massive rule when it comes to engineering management we see repeated over and over, yet no one listens: Do Not Rewrite. Period.
Apple is exemplary in this. We don't know how many changes they've made to iOS since its fork from MacOS long ago, which was based on BSD even longer ago. But have you used an iPad in recent history? Instant app starts. No lag. No-stutter rendering at 120fps. When HFS started giving them issues, they swapped it out with APFS. Apps are sandboxed completely from one-another, and have no way to break their sandbox even if the user wants them to. Etc. Comparing an iPad Pro's performance to most brand new Windows laptops is like fighting a low-orbit laser cannon with a civil war era musket. They've succeeded, however they managed to do that.
Point being, you don't rewrite. You learn, you adapt, and you iterate. We'll get there.
(And if you've read all this and then wondered "but isn't Fucshia a rewrite" you'd be right, and we should all have serious concerns about that OS ever seeing the light of day on a real product, and its quality once that happens. It won't be good. They can't even release a passable ChromeOS device [2])
[1] https://www.windowscentral.com/understanding-windows-core-os...
[2] https://www.youtube.com/watch?v=HOh6d_r63Bw
onecore isn't really about the kernel, but the intermediate layers above it.
> Would it cost more than a billion dollars to assemble an elite team of a few hundred people to build a new OS? Would Apple, Microsoft, or Google notice the billion dollar cost? Or even two billion?
It's not a matter of money, resources, or talent. For reference: The Mythical Man-Month, waterfall process, Windows Vista.
Building something of this scale and complexity from scratch will take a lot longer than even the most conservative estimate and will take many more years to shake out the bugs. Again, remember Windows Vista? And that was not nearly as revolutionary as the things you suggest.
Consider also that basically every single ground up "we are rethinking everything, and doing it right this time" OS rebuild from scratch has been a failure. There have been dozens, if not hundreds of examples.
https://en.wikipedia.org/wiki/Second-system_effect
https://en.wikipedia.org/wiki/OS/2
https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs
https://en.wikipedia.org/wiki/Copland_(operating_system)
https://en.wikipedia.org/wiki/Windows_Vista#Development_rese...
https://en.wikipedia.org/wiki/Cairo_(operating_system)
https://en.wikipedia.org/wiki/Multiuser_DOS
https://en.wikipedia.org/wiki/GNU_Hurd
https://en.wikipedia.org/wiki/BeOS
https://en.wikipedia.org/wiki/Singularity_(operating_system)
1 reply →
Google is doing something with that money: https://en.wikipedia.org/wiki/Google_Fuchsia
Good point. I forgot to mention that. I don't know if they're going for Instant Computing though. Are they?
1 reply →
I think the biggest problem is building the application ecosystem. iOS and Android were able to attract developers because they were the first movers on the computer phone platform. Convincing people to use a new OS for desktop (or any existing) computing without any apps would be difficult, and vice-versa convincing devs to support a platform with no users is also tough.
I think it's actually much easier now to get people on a new OS then it was a decade ago, or in the 90s, or the 80s. The web is so centric that a new OS with a great browser has won half the battle (I could write five pages on how massively better a browser could be than Chrome, Edge, Firefox, Safari, Opera, etc. and I love Firefox and Opera and Vivaldi and Brave.)
ChromeOS has partly proved the point, especially among college students and younger. Anyway, a serious effort at a new OS, a whole new stack, along with a new office suite and other applications, could succeed with the right (large) team and funding. People aren't very loyal to Microsoft – they're not loved. And Apple users have no idea how much better computers could be, but they would if someone showed them.
We built something like this once, AS/400, checks off many of the boxes you mentioned.
Would it cost more than a billion dollars to assemble an elite team of a few hundred people to build a new OS? Would Apple, Microsoft, or Google notice the billion dollar cost? Or even two billion?
This gives some idea of what is required to keep Linux moving forward. It would be nice if we could see similar stats for MS and Apple.
https://lwn.net/Articles/767635/
> I'd love to see a new generation of clean sheet operating systems. > We also need a fundamentally new security model.
Have you looked at Genode/seL4?
Yes, I like them a lot. A formally verified OS should be the standard these days (and formally verified compilers). But for that to be so, we'll need new programming languages (not Rust) and toolchains I guess.
7 replies →
raid1 a compresssd ramdisk to your OS drive and everything basically is instant
> The NT file system API is designed around handles, not paths. Almost any operation requires opening the file first, which can be expensive. Even things that on the Win32 level seem to be a single call (e.g. DeleteFile) actually open and close the file under the hood. One of our biggest performance optimizations for DrvFs which we did several releases ago was the introduction of a new API that allows us to query file information without having to open it first.
Ouch that sounds painful... Is this why deleting gigs worth of files takes a bit? I could of sworn it's not a huge difference on Linux, at least when using the GUI, maybe when doing straight rm it's quicker.
Reminds me of a performance optimization I did at work. A previous developer had implemented a routine to clean up old files. The routine would enumerate all the files in a directory, and then ask for the file age, and if old enough delete the file.
The problem was that asking the file age given a path caused an open/close, as GetFileSizeEx expects a handle.
Now, at least on Windows, enumerating a directory gets[1] you not just the filename, but a record containing filename, attributes, size, creation and access timestamps. So all I had to do was simply merge the directory enumeration and age checking.
The result was several orders of magnitudes faster, especially if the directory was on a network share.
[1]: https://docs.microsoft.com/en-us/windows/desktop/api/minwinb...
POSIX stuff is terrible for this sort of thing, but luckily it's often a straightforward fix once you know to look for it. Here's a similar fix for the old Win32 version of the silver searcher: https://github.com/kjk/the_silver_searcher/pull/7
Looks like the active Windows port (https://github.com/k-takata/the_silver_searcher-win32) doesn't have this change, but it's possible Mingw32 or whatever exposes this flag itself now (as it probably should have done originally).
Path-based I/O seems quite dangerous to me. If everything was path-based, you'd easily have inherent race conditions. You want to delete a directory? You stat() all the files, they look empty, so you delete them... but in between, another process writes to some file (or maybe the user forgets the file is being deleted and saves to something there), and suddenly you've deleted data you didn't expect. When you do things in a handle-based fashion, you know you're always referring to the same file (and can lock it to prevent updates, etc.), even if files are being moved around.
However, to answer your question of why removing a directory is slow... if you mean it's slow inside Explorer, a lot of it is shell-level processing (shell hooks etc.), not the file I/O itself. Especially if they're slow hooks -- e.g. if you have TortoiseGit with its cache enabled, for example, it can easily slow down deletions by a factor of 100x. But regarding the file I/O part, if it's really that slow at all, I think it's partly because the user-level API is path-based (because that's what people find easier to work with), whereas the system calls are mostly handle-based (because that's the more robust thing, as explained above... though it can also be faster, since a lot of work is already done for the handle and doesn't need to be re-performed on every access), so merely traversing the directory a/b/c requires opening and closing a, then opening and closing a/b, then opening and closing a/b/c, but even opening a/b/c requires internally processing a and b again, since they may no longer be the same things as before... this is O(n^2) in the directory depth. If you reduce it to O(n) by using the system calls and providing parent directory handles directly (NtOpenFile() with OBJECT_ATTRIBUTES->RootDirectory) then I think it should be faster and more robust.
You stat() all the files, they look empty, so you delete them... but in between, another process writes to some file (or maybe the user forgets the file is being deleted and saves to something there), and suddenly you've deleted data you didn't expect
This is fundamentally not any different between the systems, race conditions can happen either way. The user could write data to file right before the deletion recurses to the same directory and the handle-based deletion happens. Similarly the newly written data would be wiped out unintentionally.
For files for which access from different processes must be controlled explicitly there is locking. No filesystem or VFS is going to protect you from accidentally deleting stuff you're still using in another context.
5 replies →
> I could of sworn it's not a huge difference on Linux, at least when using the GUI, maybe when doing straight rm it's quicker.
Using Linux file management GUIs can be a disaster. Some years ago, I was massaging many GiB of text, using a crude hack, with scripts that did grep, sed, awk, etc. And I had many working directories with >10^4 tiny files each (maybe 10^5 even).
In terminal, everything was fine. But opening a GUI would swap out so much that the box would freeze. I don't know why. I just learned to avoid doing it.
Heh, that's fine, I don't mind the terminal or the GUI, as long as you're rationally using either and aren't blaming either for your problems. I see a lot of "I only use the terminal" type of people who screw up GUIs and the reverse can be said, GUI users who screw up the terminal, it's all about RTFM'ing sometimes, or just reading what's right in front of you.
1 reply →
Not only it's extremely surprising that there's no getting metadata without opening each file―and since DrvFs is only the WSL layer, apparently the system itself still to this day doesn't have such a feature.
But I'm now additionally baffled by how Total Commander managed to feel hella snappy ten years ago while navigating lots of dirs, whereas on MacOS all double-panel managers are meh, not least due to the rather slow navigation.
10 years ago there was less file system integration, user land virus scanning, kernel level virus scanning, os-hooks, OS-compatibility re-direction, and 32/64bit compatibility checks.
This was mostly added during NT6.0 era, which occured ~12 years ago. VISTA was the first OS using NT6.0 and VISTA was VERY much not in vogue ~12 years ago. In fact it was avoided like a plague as of 2008 (unless you were using 64bit, and had >=4GiB of RAM)
So many were using Windows-XP 32bit, or the NT5.2 kernel. Even those with >=4GiB of RAM were on Windows-XP 64bit, as VISTA had a ton of driver problems.
NT6.0 didn't catch until Windows7 and NT6.1
Thanks for sharing, that was an interesting read. I know the IO performance has always been the main gripe with WSL.
It makes me think more broadly about some of the trade offs with the discussions that happen on Github Issues.
It’s great that insights like this get pulled out in the discussions, and this could serve as excellent documentation. However, the discoverability will be difficult and it always takes some weeding through the low quality comments to piece the insights from contributors together.
I wonder how Github (and Microsoft) could better handle this. They could allow for flagging in-depth comments like this into a pinned section and they could be updated collaboratively in the future (Wiki?).
It also feels like a reputation system could help to motivate healthier discussions and could bury low quality “me too” comments and gripes. It’s not so bad in the linked example but I often encounter many rude comments aimed against volunteer OSS developers.
This particular post is pretty famous if you've been following the development of WSL. It's constantly linked to and referenced from other GitHub issues regarding WSL performance. So I think GitHub's current system is succeeding in that regard, although there are so many good points raised here that I wish it could get turned into a standalone blog post.
I can’t see MFT contention mentioned once. That’s what absolutely and totally destroys small file write performance. This affects source operations, WSL, compilers, file storage, everything.
And that’s so architecturally tied to the guts of NT you can’t fix it without pushing a new NTFS revision out. Which is risky and expensive.
Which is incidentally why no one at Microsoft even seems to mention it I suspect and just chips at trivial issues around the edges.
Bad show. NTFS is fundamentally broken. Go fix it.
Edit: my experience comes from nearly two decades of trying to squeeze the last bit of juice out of windows unsuccessfully. My conclusion is don’t bother. Ext4, ZFS and APFS are at least an order of magnitude more productive and this is a measurable gain.
Perhaps we didn't read the same article. What it says that the root of problem is the Windows IO subsystem architecture. Change NTFS for anything and you will get the same problem.
But that’s not the case. The root cause is the MFT and NTFS architecture. People fail to mention that because the problem is harder to fix. It’s almost that there is a “do not speak bad of NTFS” going on.
You can demonstrate this by using a third party file system driver for NT. when NTFS on its own is eliminated the performance is much much better. This is a neat little differential analysis which is conclusive. I can’t remember the product I used when I evaluated this about 8 years ago unfortunately.
I think this is a very good example of how windows is different in its goals and designs from Linux. I have a feeling this is the reason Linux has had a hard time catching on, on the desktop. Its easy to complain about a slow filesystem, but Microsoft lives in a different world, where other priorities exist. For someone building a server running a database serving loads of people Linux is a no brainer. You can pick the parts you want, shrink wrap it and fire it up without any fat. On a desktop machine, you want to be able to update drivers in the background without rebooting, you want virus scanners, and you want to have driver ready the moment the user plug's in a new device. Both Windows and Linux is for the most part very well engineered, but with very different priorities.
I’m very confused by your post. You start off talking about desktop machines but NT was actually engineered for servers and then later ported to the desktop. You then describe a bunch of features Linux does better than Windows (eg updating drivers without a reboot).
I think a more reasonable argument is to make is just that Windows is engineered differently to Linux. There’s definitely advantages and disadvantages to each approach but ultimately it’s a question of personal preference.
NT is engineered for a different category of servers, though - it's a workgroup server first (originally its chief competitor was NetWare), and a Web/Internet server second. That drives a different set of priorities.
For example, as someone elsewhere in the comments pointed out, NT does file access in a way that works very well when accessing network shares. That's a pretty core use case for Windows on business workstations, where it's common for people to store all the most important files they work with on a network share, for easier collaboration with other team members.
NT was architected to handle high-end workstations from day one — there’s a reason why running the GUI was mandatory even when the resource costs were fairly substantial.
Check out e.g. https://en.wikipedia.org/wiki/Windows_NT_3.1 for the history of that era. The big selling point was that your business could code against one API everywhere, rather than having DOS PCs and expensive Unix, VAX, etc. hardware which was completely different and only a few people on staff were comfortable with.
3 replies →
Drivers in Linux live in the kernel. Whenever the kernel is updated a reboot is required (in most distros). Hence your assertion that Linux updates drivers without a reboot better than windows does is questionable.
4 replies →
> On a desktop machine, you want to be able to update drivers in the background without rebooting, you want virus scanners, and you want to have driver ready the moment the user plug's in a new device.
With the exception of the virus scanner these actually sound like arguments in favour of Linux, in my experience.
(Although there are also excellent virus scanners available for Linux anyway)
I’m pretty confused by this post. What would you identify their priorities as?
Regardless, as a person not a fan of windows (not worth learning a unix alternative), I would argue it’s the polish that makes the experience worth it, not some better engineered experience. For instance: working with variable DPI seems to be trivial, whereas it still seems years off in Linux. Same with printers and notifications and internet and almost everything. These aren’t feats of engineering per se, but they do indicate forethought I deeply appreciate when I do use windows.
I would hesitate to ascribe too much purpose to everything you see. Microsoft is a huge company with conflicting political factions and a deep ethos of maintaining backwards compatibility so there are plenty of things which people didn’t anticipate ending up where they are but which are risky to change.
One big factor is the lost era under Ballmer. Stack ranking meant that the top n% of workers got bonuses and the bottom n% were fired, and management reportedly heavily favored new features over maintenance. Since the future was WinFS and touching something core like NTFS would be a compatibility risk, you really wouldn’t have an incentive to make a change without a lot of customer demand.
As a c# Dev, I am constantly annoyed that Windows updates and sometimes installs require reboots or stopping all user activity, while I've never had to reboot or block during an upgrade on ubuntu
To be fair, a lot of Linux updates require a reboot or at least a logout to properly take effect, too. Windows is just very aggressive about forcing you to upgrade and reboot, which does undeniably have security benefits when you consider that Windows has a lot of non-technical users and a huge attack surface. At least they have relaxed it a bit, the frequent forced reboots caused me some serious problems on a Windows machine I had controlling a CNC machine.
4 replies →
Most of these things are coincidental byproducts of how Windows (NT) is designed, not carefully envisioned trade offs that are what make Windows Ready for the Desktop (tm).
For some counterexamples of how those designs make things harder and more irritating, look at file locking and how essentially every Windows update forces a reboot, that is pretty damn user unfriendly.
Even without file locking, how would live updates work when processes communicate with each other and potentially share files/libraries? I feel like file locking isn't really the core problem here.
16 replies →
> On a desktop machine, you want to be able to update drivers in the background without rebooting, Exactly what does Linux and Windows DON'T does .
Windows can update many drivers without rebooting - even graphics drivers (try that with Linux and X!).
2 replies →
They mention that file performance decreases with the numbers of filters that are attached to the NTFS drive. Is there a way to list the filters to determine which ones (and how many) are applied on your system?
I had the same question. Apparently there are a couple of commands to look at:
That shows the filters loaded, and presumably the ones with "num instances" of 0 aren't actually in use on any volumes.
You can figure out what each of the filters is by looking at the output of:
To figure out what the ones I saw in fltmc were, I used this:
Interesting this article never said "poorly architected". The conclusion that the issue is peanut buttered points at that. Instead of looking into the system for things to optimize is there any proposal or initiative to rework it at a higher level?
Proposal from an earlier comment:
> Noting, as an aside, that it isn't really all that necessary for MSFT to work on the project, because I gather there are at least 684 FOSS NT kernel developers available, and qualified, and willing, to volunteer their time to work on projects like that. I assume that's why all those people upvoted, anyway. With a team that size, stabilizing WinBtrfs will take no time at all.
Windows is different. Linux puts effort into making the Unix style of IO work fast.
On similar lines, one of the usual gripe with windows is the horrible file copy estimates, Raymond Chen wrote an article on it to explain why it is[1].
[1]: https://blogs.msdn.microsoft.com/oldnewthing/20040106-00/?p=...
The Old New Thing is a fantastic blog especially if you develop in Windows. I used to read it religiously until 5 years back and stopped reading about time when Google Reader was retired and somehow didn't setup in my new flow of reading blogs. Thanks for reminding me about this blog and article again.
At least windows doesn't freeze your whole desktop under heavy IO.
Edit: I'm getting downvoted by people that never used Linux probably.
You're being downvoted, but it's true about linux for at least two reasons to this day:
1. dm-crypt threads are unfair to the rest of the system's processes [1]. On dmcrypt systems, regular user processes can effectively raise their process scheduling priority in a multithreaded fashion by generating heavy IO on dmcrypt storage.
2. Under memory pressure, the VM system in linux will enter a thrashing state even when there is no swap configured at all. I don't have a reference on hand, but it's been discussed on lkml multiple times without solution. I suspect the recent PSI changes are intended as a step towards a solution though. What happens is clean, file-backed pages for things like shared libraries and executable programs become a thrashing set resembling anonymous memory swapping under memory pressure. As various processes get scheduled, they access pages which were recently discarded from the page cache as evictable due to their clean file-backed status when other processes ran under pressure, and now must be read back in from the backing store. This ping-ponging continues dragging everything down until either an OOM occurs or pressure is otherwise relieved. This often manifests as a pausing/freezing desktop with the disk activity light blazing, and it's only made worse by the aforementioned dmcrypt problem if these files reside on such volumes.
[1] https://bugzilla.kernel.org/show_bug.cgi?id=199857
This behaviour - and perhaps other causes with similar issues for desktop users, is what drove me away from helping to make Linux ‘ready for the desktop’. The kernel philosophy was incompatible with the needs of desktop users. Those developing for the desktop (like myself) didn’t have the required expertise to make the kernel do the Right a Thing, and couldn’t find enough people willing or able to help.
Things may have changed over the years - I’ve been running a Linux desktop recently and haven’t seen this kind of issue yet (the kind where you need to use magic keys to ask the kernel to kill all, sync and reboot) but reading your post, perhaps this is because RAM is much more plentiful these days.
1 reply →
Yeah, I still observe this from time to time, with SSD-only storage and 32 GB main memory.
1 reply →
Is there no way to set a minimum page cache size?
It does?
It does even much weirder things. https://blogs.technet.microsoft.com/markrussinovich/2007/08/...
Oh my God, they pretended everybody has a 100 Mbps NIC and capped at that speed. That's completely retarded.
At least I'm going to give them credit for disclosing what the problem was. I would've be too ashamed to admit it.
I work and play every day over Linux and I never saw the desktop freezing.
I’ve attempted to use Linux and desktop freezing is the norm even on machines that run fine under windows. Admittedly the machines might be underpowered but that does call into question the commonly held belief that desktop Linux is better for low spec machines.
I have had issues at times in the past but with things like core dumps on systems with spinning disk and 128gb ram. The OOM on linux can be brutally frustrating. But it's still light years ahead of windows for development...
I haven't noticed Linux doing that in the last few years.
But honestly, that could also be the NVMe drives.
Depends entirely on your choice of I/O Scheduler.
With bfq-mq, I've encountered no desktop freezing under heavy I/O.
Do any mainstream distro use this?
> At least windows doesn't freeze your whole desktop under heavy IO.
It does it as well on Windows and unlike Linux, Windows still has a risk of permanent damage under low space available, it was even worse in the XP days but these issues are still there.
You're probably getting downvoted by people who have used Linux in the past 10 years.
It was / still is that the default scheduler is / was bad under heavy IO, copying a file on a USB disk would freeze the whole system ( on desktop ).
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/131094
6 replies →
I'm on Linux right now and can cause this to happen by using more than 60% of my RAM. Afaik no distros correctly handle swapping onto a HDD.
If I hit 80%, I get 10-20 second lock ups.
If I hit >95%, I get 1-2 minute lock ups.
Using Ubuntu 18.04.
11 replies →
I've experienced a total freeze before once one of my programs started swapping/thrashing. The entire desktop froze, not just the one program. This was in the past two years or so, so it's not a solved problem.
my experience with windows is that it freezes the whole desktop under moderately multithreaded loads.
It doesn't? Oh right, you're bullshitting me.
Case in point: I recently tried unzipping the Boost library on an up-to-date Windows 10, and after trying to move the frozen progress window after a minute, the whole desktop promptly crashed. I have to say, the experience is better than it used to be, because at least the taskbar reappeared on its own. (Decompression succeeded on the second attempt after leaving the whole computer well alone ... but it certainly took its time even on a high-end desktop computer.)
Could you please leave personal swipes out of your comments here? They have a degrading effect on discussion and evoke worse from others. Your comment would be just fine without the second sentence.
https://news.ycombinator.com/newsguidelines.html
The issue seems to be with how Windows handles file system operations. They allow filters to operate over each request. These filters are like hardware drivers in that they are created and maintained by third parties. So, MS has no real ability to change how things work because they don't have absolute control in the same way that Linux does over the operations (Linux device drivers are part of the kernel)
Not sure if I'm being naive, but couldn't they make it so that if there are no filters present on a given volume, then the entire filter stack is bypassed and the directory nodes are served from a cache managed by the file system at the root of the volume? That way developers who really want performance can make sure they have no filter drivers installed on that volume.
The linked post notes that, even on a default Windows install, you already have a sizable stack of filter drivers active. Not sure whether that only applies to the C: volume or to other ones as well.
If they really wanted to, they could mount an ext4 fs as a single blob on disk directly into WSL. That won't really help if you want to run a GUI editor in windows against that identical directory though.
Any idea why they haven't responded to people's requests for FUSE? It's the second-most requested feature and should enable applications like this.
3 replies →
NTOS (Windows NT's kernel) was originally designed to be backward compatible with Windows (DOS era), OS/2 and POSIX. Designing such a kernel was considered a performance at the time but probably cost its fair share of bloat that kept growing with the years. Also, it's not surprising that Linux is optimized for operating on files since the UNIX philosophy is about considering everything as a file, something that Dave Cutler (NTOS main designer) was known for criticizing[1].
I pray daily that more of my fellow-programmers may find the means of freeing themselves from the curse of compatibility. (Dijkstra)
[1] https://en.wikipedia.org/wiki/Dave_Cutler#Attitude_towards_U...
As another data point here: I had to add a TKinter-based loading screen to a PyQt5 app I wrote for work because the app - which takes half a second to start up on Linux - takes nearly a minute on Windows the first time around (and multiple dozens of seconds on subsequent runs) and I know firsthand that my end-users are unlikely to wait that long unless they can see something loading. I suspect it has to do with the sheer number of file accesses for each of the Python libraries and underlying DLLs and such.
> A Win32 path like C:\dir\file gets translated to an NT path like \??\C:\dir\file, where \??\C: is a symlink in Object Manager to a device object like \Device\HarddiskVolume4
Cool! Can the "NT paths" be used directly on Windows? As far as I know the ordinary Win32/NTFS system doesn't even have symlinks and this feels quite a handicap.
New-Item -Path C:\LinkDir -ItemType SymbolicLink -Value F:\RealDir
Note that this is behind a "developper flag", i.e. it needs elevation if you have not checked "developper mode" in the W10 parameters. (I guess they don't want the cost of supporting ordinary people building circular dependencies)
It depends on which API the application uses to pass paths to the OS and whether its path handling library does internal parsing that is or is not aware of those prefixes.
For example rust has support[0] for the magical file path prefixes[1] that can be used to escape path length restrictions, reserved identifiers and some other limitations. This is used by default whenever you normalize a part, which means it has fewer restrictions than windows explorer has by default.
[0] https://doc.rust-lang.org/std/path/enum.Prefix.html [1] https://docs.microsoft.com/en-us/windows/desktop/fileio/nami...
Windows does have symlinks and file junctions. Not sure how these compare to other file systems.
These require user-mode round trips for path traversal, so comparable to FUSE, I believe.
So it is relatively easy to add hooks of your own that get called as if they were file reads; NTFS calls these "Reparse Points". Linux would do this with named pipes.
I am just a user of these things, haven't dug into any implementation details recently, but I guess that the stacks to support these features in Linux and Windows each have their cache etc design decisions that have led to current state of filesystem performance under typical use-cases these days.
I'm not a super expert but I think that is the case and you even need them if you want to use some advanced features like paths that are longer than MAX_PATH or these virtual mount kinds of things.
But IMO it's ugly as hell and just begging for compatibility problems. So don't go there, at least not to get super long paths. 260 bytes should be enough for anybody.
> 260 bytes should be enough for anybody
It happens to be not enough just so damn often.
10 replies →
Yes they can, though one problems is that because NT paths are less restrictive than Win32 paths, you can create files / directories that you then have trouble doing things with using standard tools
Notably the Windows shell does not support longlong file names, like, at all. The Windows shell and Win32 APIs are also easily confused by files using the NTFS POSIX namespace, which may have \ in their name.
I still scracth my head as to why doing a kernel update of Ubuntu running under Hyper-V on a spinning disk is so horrifically slow. If I migrate the VM while it's running to SSD, let it finish, then migrate it back to spinning, it's faster then just letting it update in place. This is my device desktop, otherwise I would put in on SSD full time.
The above comments make me think something like the stat calls may be the issue and moving the running vm to SSD hides the problem. It obviously isn't raw disk rate at the point.
Is the Ubuntu VM's / on a fixed or dynamic VHD? The former provides (modulo filesystem fragmentation) linear addressing of virtual LBAs to physical LBAs, and the latter can be heavily fragmented and result in sequential read/write operations becoming random access, which on a HDD kills performance.
My advice for anyone running VMs is if they're on HDDs or they're highly tuned for disk usage (databases for example), use fixed VHDs, otherwise use dynamic.
Thanks for the suggestion. The disk is a dynamic VHDX, migrated it to fixed vhdx and will experiment.
I still scracth my head as to why doing a kernel update of Ubuntu running under Hyper-V on a spinning disk is so horrifically slow.
Define slow please.
For a laugh I picked a random VM (VMWare) at work and ran (I did apt update first):
The updates included two client server DB engines (Clickhouse and Postgresql) the fileserving thingie (Samba) and a few other bits. The reboot takes about 90 seconds before the webby interface appears for rspamd.
Specifics,Host OS is Windows 10 Pro,virtual machine is Hyper-V, Guest OS is Ubuntu, Dynamic VHXD, Storage Space Pool with 4 HDD and 1 SSD, Windows Filesystem ReFS.
Only observe this specifically with kernel updates. Everything updates as I would expect on a HDD.
Slow is ~10-20 minutes. Which is why it is faster to migrate the running VM from the Storage pool, onto a single SSD, complete the ubuntu kernel update and migrate back.
Have you cleaned away the old kernel packages? Kernel package operations on Debian and Ubuntu are accidentally quadratic with a high time constant (there is a package script for each kernel that iterates over all installed kernels) so you want to make sure not to let them accumulate.
Has to be some kind of caching issue ...
The build speed difference for Android (Gradle build system) between Windows and Linux is extremely noticeable especially for large projects, where you can see even almost 50% faster builds.
Wouldn't holding project files in an ext partition fix this? Or is that not supported yet?
Use ReactOS driver if that works. If not, file a bug report on https://jira.reactos.org.
Keep in mind that Linux filesystems are all implemented in the kernel (FUSE aside), and WSL doesn't run a Linux kernel - it just emulates syscalls for userland to work. So there's no ext drivers in there, or any other standard Linux FS drivers.
The way to really fix it would be a filesystem on a loopback device that stays inside linux territory, but that's not supported yet.
AFAIK Windows 10 doesn't support extN (N = {2,3,4}) partitions
The entire post negates the first statement that it isn't easy to just say "NTFS is slow".
I would love a QT-esque rust library that utilizes components and maybe even scss or css for styling
tldr: discussion is about why disk access under WSL is slower than under Linux, mostly due to the different design constraints of Linux vs Windows for file systems.
An interesting comment from insider. We all knows from various benchmarks that Microsoft Windows' filesystem access performance was way too worse than Linux kernel not only WSL but also Win32 Subsystem too.
Also, process creating performance is worse on Windows. I wonder if it is also the case of "Death by thousand cuts"
Windows performance is abysmal. Instead of beating around the bush, they should just state that it is the combination of everything that makes Windows dog slow compared to either Mac or Linux. Someone said it on that thread, but Windows really should be a DE on Linux now instead of being its own OS.
Perhaps this explains why Yarn/NPM was slow on WSL too?
dyt