Assuming you have a route enabled on your vNet to send traffic to the Azure Firewall then you won't be able to RDP directly to the Public IP of your VM. The problem here is something called asymmetric routing. Your inbound request goes to the public IP of the VM, the RDP client expects the response to come from the same IP, however because your outbound traffic is being routed to the firewall, the response comes from the firewall IP. The client does not allow this, as it is a security issue, so your connection fails.
To resolve this you either need to use a NAT rule on the FW to have the inbound and outbound request go via the firewall, or exclude the VM from the route that sends traffic to the FW (but this obviously means traffic from that VM is not monitored). You could also look at using Azure Bastion for your RDP connection.