I didn't want to start a new thread so I'm just gonna publish my findings here.
I have an LG 27UD68 4K monitor connected directly to my Surface Pro 4 via a DisplayPort cable. For some reason wifi is extremely slow in DisplayPort 1.2 mode, but in 1.1 mode it works just fine. This leads me to believe that this is not a hardware or an interference issue. From time to time the wifi will start working completely fine, for up to 30 minutes, and will randomly start working very slowly again (for example in the middle of streaming something, so absolutely no touching the computer).