Comment by leptons

>but the iframe still gets loaded when it enters the viewport even if the user has no intention of watching the video

That doesn't affect page speed scores if the video is "below the fold", and that's all that I really care about. If Google Lighthouse doesn't complain about it, then my job is done.