Comment by jonplackett

Yes, and it's allowing the format to change to allow more cleverness or apply more processing power.

For example, changes from one frame to the next are encoded in rectangular areas called "superblocks" (similar to a https://en.wikipedia.org/wiki/Macroblock). You can "move" the blocks (warp them), define their change in terms of other parts of the same frame (intra-frame prediction) or by referencing previous frames (inter-frame prediction), and so on... but you have to do it within a block, as that's the basic element of the encoding.

The more tightly you can define blocks around the areas that are actually changing from frame to frame, the better. Also, it takes data to describe where these blocks are, so there are special limitations on how blocks are defined, to minimise how many bits are needed to describe them.

AV2 now lets you define blocks differently, which makes it easier to fit them around the areas of the frame that are changing. It has also doubled the size of the largest block, so if you have some really big movement on screen, it takes fewer blocks to encode that.

That's just one change, the headline improvement comes from all the different changes, but this is an important one.

There is new cleverness in the encoders, but they need to be given the tools to express that cleverness -- new agreement about what types of transforms, predictions, etc. are allowed and can be encoded in the bitstream.

https://youtu.be/Se8E_SUlU3w?t=242

I believe patents play a big role here as well. Anything new must be careful to not (accidentally) violate any active patent, so there might be some tricks that can't currently be used for AV1/AV2

A bit of both. Also, the modern Codecs have slightly different tradeoffs (image quality (PSNR, SSIM), computational complexity (CPU vs DSP vs Memory), storage requirements, bit rate) and therefore there isn't one that is best for every use case.

I wonder when we will see generative AI codecs in production. The concept seems simple enough, the encoder knows the exact model the decoder will use to generate the final image starting from a handful of pixels, and optimizes towards lowest bitrate and minimum subjective quality loss, for example, by letting the decoder generate a random human face in the crowd, or give it more data in that area to steer it towards the face of the team maskot, as the case may be.

At the absolute compression limit, it's no longer video, but a machine description of the scene conceptually equivalent to a textual script.

New video codecs typically offer more options for how to represent the current frame in terms of other frames. That typically means more processing for the encoder, because it can check for all the similarities to see what works best; there's also harder math for arithmetic coding of the picture data. It will be more work for the encoder if it needs to keep more reference images and especially if it needs to do harder transformations, or if arithemetic decoding gets harder.

Clever matters a lot more for encoding. If you can determine good ways to figure out the motion information without trying them all, that gets you faster encoding speed. Decoding doesn't tend to have as much room for cleverness; the stream says to calculate the output from specific data, so you need to do that.

I don’t know the details of AV2, but going from h.265 to h.266, the number of angles for angular prediction doubled, they added a tool to predict chroma from luma, added the ability to do pixel block copies and a bunch of other techniques… And that’s just for intra predictions. They also added tons of new inter prediction techniques.

All of this requires a significant amount of extra logic gates/silicon area for hardware decoders, but the bit rate reduction is worth it.

For CPU decoders, the additional computational load is not so bad.

The real additional cost is for encoding because there’s more prediction tools to choose from for optimal compression. That’s why Google only does AV1 encoding for videos that are very popular: it doesn’t make sense to do it on videos that are seen by few.

It’s more money and more user’s compute being thrown at the problem to get the streaming service’s CDN bill down.

Soon we will just have local AI processors which will just make stuff up between scenes but adhere to a “close enough” guideline where all narratively critical elements are maintained but other things (eg landscapes or trees) will be generated locally. Movies will practically be long cutscenes with photorealistic graphics.