To allow User X to connect via RDP only to Computer Y, you need to control two things: where the user is allowed to log on through Remote Desktop Services, and ensure they are denied everywhere else.
First, on Computer Y, make sure User X is allowed to log on via RDP. In Group Policy, create or edit a GPO that applies only to Computer Y (link it to the OU containing Computer Y). Go to Computer Configuration → Policies → Windows Settings → Security Settings → Local Policies → User Rights Assignment. Configure “Allow log on through Remote Desktop Services” and add either User X directly or, preferably, a security group that contains User X. Also ensure User X is not listed in “Deny log on through Remote Desktop Services” in that same GPO.
Next, prevent User X from connecting to any other computers. Create another GPO that applies to all other computers (for example, link it at the domain level or to the relevant OUs, excluding Computer Y). In that GPO, go to Computer Configuration → Policies → Windows Settings → Security Settings → Local Policies → User Rights Assignment and configure “Deny log on through Remote Desktop Services” to include User X (or the group containing User X). Because deny rights override allow rights, make sure this deny GPO does not apply to Computer Y. You can exclude Computer Y either by placing it in a separate OU and not linking the deny GPO there, or by using security filtering so the deny GPO does not apply to Computer Y.
A clean design is to create a group such as “RDP-Only-ComputerY”, add User X to it, allow that group in the “Allow log on through Remote Desktop Services” setting on Computer Y, and configure a domain-wide GPO that denies that same group the “Log on through Remote Desktop Services” right, excluding Computer Y from that deny GPO scope. After configuring, run gpupdate /force on the target machines and verify effective rights with rsop.msc or gpresult /h report.html.
If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.
hth
Marcin