Hi Nicholas,
I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this!
Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.
Issue:
Websocket connection to xxxx failed for React Static Web App + Django App Service application
Solution:
After unlinking my app service from my static web app, I was able to finally get the websocket connection through.
Please click Accept Answer and kindly upvote it so that other people who faces similar issue may get benefitted from it.