Comment by moron4hire

There are so many ways in which the desktop experience could be improved, exactly because, as Jensen pointed out, it has basically stagnated since the 90s.

You could basically look at all the ways that UX has advanced on the Web and almost generate a very long TODO list of things to improve on the desktop. I know there are a lot of people here, especially of the Linux using variety, who revile Web UX, but the things most people complain about (privacy concerns, bloat, document vs app platform) happened specifically because desktop UX stopped innovating 30 years ago.

Some personal peeves, all from having to reimplement these things in projects over the years, which is now many projects because I'm now old, so having to do the same things over and over again is getting annoying:

Multiple Document Interfaces used to have built-in support in GUI toolkits. Pro-oriented editing environments like Visual Studio not-Code and Photoshop gave you the ability to have multiple documents open and operate on then on a single application context. While this is a bespoke implementation in those programs now, that's largely because the UI toolkits didn't keep up with newer features like docking, tabbing, etc. Browsers themselves also had to implement bespoke UI for tabbed documents, again for the same reasons. It's shocking how bad this is in VSCode, where certain kinds of sub-Windows can only be put in certain places, but again, that's because they had to do it from scratch. Now, there is no MDI support in the browser at all, you're left implementing it from scratch, and modern GUI toolkits seem to all just try to ape HTML/CSS (not that I think that's a completely terrible approach, but only because The Desktop Toolkit is Stagnant).

With modern large screens, I find myself at a loss for how to use all the screen real estate effectively. Not that I lack for need, but because I can't seem to get everything I want to see into a comfortable position without sacrificing some other thing. Yet, at the same time, I can't seem to fit much in my own application window. This one is more on me for not having too great of design skills, but I think there are some fundamentally difficult issues to solve here since nobody is going to have a 40” laptop. Fundamentally, the issue is one of being too rigidly reliant on the Application Developer to implement such things as MDI and tool layout. We kind of figured that out with responsive designs break-pointing when going to truly small screens on phones, but I think this could be done more automatically if the GUI toolkit was better.

Smartphones, laptops, and desktops aren't the only 3 workspace sizes. There's also the completely borderless, 3D space of the VR home environment. After experiencing Windows Holographic for the first time and experimenting with using a Hololens as a primary compute device (and then drinking away the massive headache), I found a lot of interesting glimmers of how spatial computing could do a lot for people. I really enjoyed having 2D documents mounted on the wall behind my desk, not needing a screen to take up space on my desk. Having a music player app pinned on a specific spot like a HiFi stereo stack just felt so much more comfortable than wearing headphones, and it was so much easier to turn my body to interact with the stereo rather than sift through my window stack to find the music app. Keeping certain app widgets in certain locations around my house (notes on my refrigerator door, Email away from coding) felt very natural and like it was helping me re-establish balance in my work. But it was all for naught as MS killed Windows Holographic and all the other devices are just pushing virtual 2D screens on which unmodified WIMP interface desktop or mobile apps can run.

Handling the myriad of document types bespoke for every application is extremely bothersome. Having only the file extension for any hint at the content type is extremely unreliable and incomplete (there are thousands of known content types not just image/png, image/jpeg, text/plain, text/markdown, and application/json). Plus, the extension only tells you the container format, the data itself could be in wildly different schemas/encodings, especially when dealing with .json or any audio/video files. Doing deep content inspection is slow and also unreliable because so many JSON-based formats don't reliably implement JSON Schema. This could be solved if we got rid of file extensions and had the full content type string embedded with the file at the File System level. A file's name should be a description of the contained content, not overloaded to store data encoding information.

After writing all of this, it started to dawn on me that these are all the same problem of wanting to composite application together from smaller bits in a unified workspace that I the User control, not dependant on me the Application Developer to provide the support for it. Because, at the application layer, I can't. I can't unify whatever features I'm building with features other app devs have built. We need this to be at the DE level or there isn't a standard to which app devs can work.

Even my complaint about content types fits into this because most of the time I just need to display and play back a file in whatever the one standard way there is to display such a thing. I'd much rather the UI toolkit provided better media players with an OS that went beyond the application window for giving people the ability to composite functionality together. With Adobe Acrobat installed, I as a User should now be able to use PDFs anywhere else I want to view Text or Markdown or Word documents and I as an Application Developer should not have to think about what document formats my users might throw into my application.

The VR home environment makes this issue a lot easier to see, as the Window does not make sense there, but I think this generalizes to 2D, screen-bordered scenarios. Most of the need for better application composition is around not having to rely on every Application Developer to implement everything themselves. Because, when you leave it to App Devs, you get abominations like Snapchat and Instagram.