Hello KingGamecube,
Welcome to Microsoft Q&A forum.
This seems like an issue with the mapping or streaming signal with your Xbox PC app. Before we proceed further, may I know if the issue with controller appears when you play a game directly from the computer? (Xbox app games is preferred for testing purpose).
Also, can you please try the website Remote Play feature instead? It is a newer feature compare to the older workflow of the Xbox app. You can access it by going to https://xbox.com/remoteplay.
I'm waiting for your reply.