I finally figured this out! :-D
I was reading over a similar issue (https://answers.microsoft.com/en-us/windows/forum/all/videos-on-all-media-players-and-browsers-keep/2d1027c2-58f0-49d8-b01d-653d895ad99c?correlationid=0fb4762a-3b41-40e6-8319-6bb480407d33&from=ContactUsWebBCQR) where someone was encountering a similar problem and they tracked it down to a Bluetooth connection. In my case, my computers are connected to a Bluetooth speaker in my room, but so are some other devices (phone, tablet, etc). This problem happens when something else is playing to the Bluetooth speaker. Windows 11 realizes it can't play the audio, so it immediately pauses the video.
It's incredibly non-obvious that this is happening, but once you know this, it's easy to fix (stop the other audio playback or disconnect from the speaker).
I still think this is a Windows bug - video playback should not pause entirely just because the speaker is in use.