When you connect to the public firewall IP you are using public networking. When you want to connect to the private IP, the computer you connect from needs to have a private connection to the network containing your VM. Azure VPN Gateways and ExpressRoute are two ways to create a private network between an on premises location (Site-2-Site VPN or ExpressRoute) or home and mobile locations (Point-2-Site VPN).
If you want to connect to your VM's without exposing them on a public IP through the firewall, Azure Bastion provides an excellent means of home- and on-premise access to your VM's with all the security mechanisms like MFA and Azure RBAC for security through terminals in the Azure Portal.