I typically just specify the height of the video and let the browser figure out the width and aspect ratio. The most annoying layout shift is the vertical kind anyway, so that solves it to my satisfaction.

That said, I also use the poster feature of the video tag and set preload to none, this produces vastly faster page loading, as images are a fast-path compared to browsers loading a video chunk and then decoding it just to display a cover image. I have a set of scripts that generate the poster images for me, I just specify the frame number I want to use in the video and ffmpeg produces an avif.