Okay, I've figured it out. It had nothing to do with swapping slots and was simply reproduced by accessing through a custom domain name (direct azurewebsites.net access worked fine)
I had:
app.UseHsts();
app.UseHttpsRedirection();
But because I'm using Cloudflare's HTTPS certificates which only does from browser to cloudflare, Azure was receiving an incoming HTTP connection and attempting to redirect to HTTPS, when it was already HTTPS.
Not sure why this behaviour didn't happen in .Net Core 2.2, but in .Net Core 3.1 it definitely does!
I've removed this for now till I upgrade and use my own certificate and this has resolved the endless redirect loop