According to this article there is a default certificate policy assigned to key vault certificates which can be altered for newer versions of existing certificates
I tried updating the KeyNotExportable setting (part of this command) for an existing certificate and it had no effect, newer certificates were still exporting with the private key.
I also tried updating certificate from the UI (Key vault -> Certificates -> Issuance Policy -> Advanced Policy Configuration -> Exportable Private Key) and it had no effect on newer certificates as well.
My indication is based on being able to view encrypted private key on exported certificate (using this command openssl pkcs12 -info -nocerts -in MY_PFX.pfx)
There was a way for me to see that it was actually working when using the cli command az keyvault certificate import which probably creates a policy starting with the first certificate
Can you verify the required behaviour?