If I understand corrected, you have an AKV which allows access from VNET and also from certain IP addresses, isn't it?
AKV can allow access from a VNET, and it also has a firewall where you can allow IP addresses directly from the resource itself.
In your first case, the access is routed from both VNET and also from the IP addresses and that's why you were able to access it.
In the second case when you remove the IP address then the access to AKV is allowed only from the VNET.
I'm not sure how this VNET is configured, but as long as you access the key vault from a resource within the VNET, then it should work fine.