Gee wiz, I don't think so or at least it will likely be painfully difficult.
Windows 11 is all about that security so Windows 11 connecting to Windows XP with Windows application would be a major security risk especially RDP and so that's not likely. I know your situation is legit but applications won't even be compatible with security anymore nor the connection.
Edit: NLA if Windows 11 has a setting for disabling that would at the very least be required to get it to RDP to XP since XP doesn't have that security. Warning: you really shouldn't disable it:
Even Steam, which blows Windows RDP out of the water, isn't even supported for Windows 7 anymore let alone XP and that old XP version most probably won't work on Windows 11.
Jump Desktop RDP only supports Windows 7 and newer so that won't work either.
Etc.
Alternatives:
You'd have to find a way to get a compatible connection between the two. Even external hardware would make that difficult. There might be something out there lol.
Oh I know. Have a video camera on it and a human controlling the computer and tell them what to do over a mic. It will work for RDP :-D
If your software would run on Linux using Wine however then you could just RDP from Windows 11 to Linux. That would have to be tested by you though since its in house software. No way of knowing if that specific software would work since Wine is crazy hit and miss. There's also a steep learning curve.
Or if that computer and software will work well enough on Windows 7 then you might find a solution for another year or two or so by installing Windows 7 on there.
But if an RDP software exists that will work on Windows XP and 11? That might be quite painful.