Comment by torginus
16 hours ago
One thing shared between old software rendered desktops and modern videogames is that they shunned these intermediate 'composited' screens. On the desktop, there used to be the screen, and basically windows used to draw on top of each other, there was no intermediate buffer for the whole window to draw itself to, which would then be smushed together with all others.
Video games are the same (mostly) - everything is rendered in screen space for performance reasons, it's very, very rare, that you would render something into a temporary buffer then composite it on top of the rest of the scene - you would need exceptional reasons for that.
Maybe it's time to get back to the olden days of display servers - where applications would push a list of render commands to the 'display server', which would consist of rendering primitives, which would then take these commands and construct the whole UI on the screen, without the intermediate steps of each app drawing into its own little buffer.
You could always fall back to drawing your own applciations, then asking the display server to composite that, but that would pretty much be the exception, not the norm.
There’s a reason why all modern desktop environments are designed the same way: power efficiency when multitasking.
Imagine you have 3 windows visible at the same time: a videogame rendering at the refresh rate of the display 144 Hz, a video player rendering frames at 30 Hz, and a text editor rendering blinking cursor at 2 Hz. Because the videogame wants to deliver frames at 144 Hz, the desktop compositor has to deliver the entire desktop at 144 Hz. Asking the video player and especially the text editor to also deliver frames at that frequency would be wasteful. Irrelevant for desktops with fast discrete GPUs, but directly translates to battery drain on laptops.
I don't think this is any better for power efficiency? You're taking the modern system as an axiom, which it isn't. The old system didn't "deliver frames at" any specific rate. The graphics chipset sent to the monitor whatever was in the framebuffer at a specific rate, and windows updated whenever they wanted to. The text editor didn't "deliver frames at 144Hz" or any other Hz - it updated some of the pixels in the framebuffer when you pressed a key.
It cost nothing to not change the pixels when you didn't press a key, no matter whether you weren't pressing keys at 60Hz or you weren't pressing them at 144Hz.
Most games haven't rendered directly into the "screen buffer" for 15-20 years.
Vast majority of titles use deferred rendering, and lighting is done off screen too. Usually the only thing done to the "screen buffer" is a final post-process pass or a copy.
Deferred renderers work differently from compositors. They still build up the entire displayed frame in screen space, except they dont write into a color buffer like directly, but produce a bunch or intermediate buffers, called G-buffers. Then they have a postprocess pass when they resolve these buffers into the final image.
This is called 'compositing' but its similar in name only. It's a fairly efficient process where each color pixel is produced by reading these buffer targets and producing a final color in a shader.
This is entirely different from what composited apps do, where they build up the app's background into a texture, and push that onto the screen, with potentially multiple screen's worth of windows living in memory. This would be equivalent in video game terms to rendering every character and object in the level as 'stickers' and then making the final image of these cutouts, which would consume tons of RAM uselessly, and would force us to render crazy amounts of detail that would never get shown.
It's one possible rendering pipeline. Is it really most games?
Regardless, video games normally update the entire screen (or window) every frame, because the screen is so dynamic. This is unlike Microsoft Excel which has a mostly static screen. Building Excel as if it's a video game is going to waste resources.
This would really help responsiveness. Keypress to pixel is a huge issue in these bloated frameworks which are running on top of bloated OS