You can try couple of things here.
Can you log into any VMSS instance and check if the nginx is installed? If yes, can you try to access it locally ?
If you are able to open the site locally, then this there are no issue with the custom script.
Then you will need to check on the NSG to see if port 80 is opened to the IP from where you are trying to access.
If you the nginx is not installed correctly, and if that is an issue with the custom script, we need more logs to troubleshoot that.
If you want to RDP to the vmss instance, you will need to configure NAT rules on the Load Balancer for each instance. Then you can use the load balancer ip and the port you have created in NAT rule to RDP to the instance.
Are you able to RDP to the instance ?