Typically a lack of bi-directional connectivity is due to either a block in a firewall / NSG, or an issue with the VPN configuration. We know that traffic can flow because you can establish a TCP connection one way, so I would start with checking Firewalls or NSGs.
I would start by making sure your on-premise firewalls and Azure NSGs are not blocking the traffic. NSGs can sit on both the NIC of the VM, as well as the subnet that the NIC is in. You can view the effective rules by going to the 'networking' tab of the VM in the Portal. Make sure that traffic is allowed outbound on port 22 to your on-premise IP Range. Also check any on-premise firewalls that could be blocking the traffic on port 22 inbound.
Here is a good TSG on VPN Gateways. Not every step is applicable to your situation, but it will be worth it to go through and double check your configuration.