Hello @Matheus PD
Yes, you can restrict access to your RDS server.
Login into RDS server itself > Windows Key+R > systm.cpl > Remote > Remote Desktop > Select Users > Add as appropriate.
If you have RDS Session Collection please specify appropriate User Group that should have access to your RDS server
https://learn.microsoft.com/en-us/windows-server/remote/remote-desktop-services/rds-user-management