Thank you for asking this question on the **Microsoft Q&A Platform. **
When you connect from a Private IP to a Public IP, the NSG only "sees" your Public IP.
If you need to block access to the host, you must change the private IP for your public IP in the NSG
You can create rules for private IPs when you are via VPN or in other VLANs
Hope this helps!
----------
Accept Answer and Upvote, if any of the above helped, this thread can help others in the community looking for remediation for similar issues.
NOTE: To answer you as quickly as possible, please mention me in your reply.