Hello ,
The remove command won't work until atleast one Access and network configuration is defined. As per Terraform The Default Action to use when no rules match from ip_rules / virtual_network_subnet_ids. Possible values are Allow and Deny.
Solution: Add the subnet of network from which you are accessing the keyvault.
Thanks,
Akshay Kaushik
Please "Accept the answer" (Yes/No), and share your feedback if the suggestion works as per your business need. This will help us and others in the community as well.