@LAUWER I like your idea in regards to using custom domain, however, I'm concerned that M365 anti-spam engine will see that it resolves to the *.5 and quarantine it anyway. I've not tested it, so maybe it will work fine.
In terms of a solution, I was thinking you could open an Azure support request under Static Web Apps (SWA) service and put pressure on them to investigate internally with M365 team. Similarly, you could open support request on M365 side as well, inquiring why a new Static Web App is being quarantined when it has no prior history of being a phishing site (or even existing for that matter).
Making progress with the support case(s) will likely take some patience as I would guess at the lower levels they will potentially want to shift blame and say their service is operating fine so there is nothing they can do. This is where polite pushback will hopefully get them to escalate further.
One question for them is, Are they okay with a large swath of SWA FQDNs being "blacklisted" by default, by their own malware detection mechanism?
On separate note is what is happening with your local ISP firewall/modem/router device. I don't know what is going on with that, if it is looking to some sort of shared blacklist database, or what? And if it is, why is it causing strange error instead of simply blocking the connection to the site altogether? I think this might require taking in-depth look at the specific hardware device and its configuration options, and local packet capture and analysis.
Maybe updating the ISP firewall/modem to latest firmware will magically "fix" the SSL_ERROR_RX_RECORD_TOO_LONG error. Worth a try.
If the above is an acceptable "Answer" to you I will convert it to answer so you can accept it, otherwise if it isn't acceptable (I understand if that is how you feel) I will leave it as a comment. Maybe someone else can chime in here and give new/better insight.
Thanks for reading this far.