Hi @Krzysztof Białogłowy,
Based on your query, I understand that DevOps stopped working once you have enabled firewall for key vault.
As per this information, you may need to use self-hosted agent in your Azure tenant as VM or scale-set, then you can configure firewall and even use KV in private network. You can configure scale set using the following document: Create the scale set. This scale set helps you in having agent on a VM.
I hope this information is helpful. Please feel free to reach out if you have any further questions.