VM and recovery services vault with CMK, snapshot for backup fails

Question

VM and recovery services vault with CMK, snapshot for backup fails

Oliver Schwarzwälder 0

Hey Guys,

i'm facing the issue that my Azure backups are failing when using a cmk encrypted recovery services vault.
-> by turning back to a PMK rsv everything works fine.

-> the received error messages in azure are in the sub task

Take Snapshot: in progress -> failed

-> the backup Job runs along for hours till ending in failed

I viewed the eventlogs on the machine and saw that there are issues with the VSS Services permissions while handling the snapshot -> why does the Azure Backup Agent runs into trouble when my rsv is cmk encrypted?

I tried also to set permissions for the recovery services vault to access with administrative roles, still facing the same error

Screenshot 2023-10-04 at 17.13.13.png

SadiqhAhmed-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-05T03:41:26.3233333+00:00

@Oliver Schwarzwälder Thank you for reaching out to us on Microsoft Q&A platform. Happy to assist you!

I understand that you are facing issues when using a CMK encrypted recovery services vault.

Encryption of backup data using customer-managed keys - Azure Backup | Microsoft Learn

Based on the move back and forth, going to guess this is relevant:

This feature allows you to encrypt new Recovery Services vaults only. Any vaults containing existing items registered or attempted to be registered to it aren't supported.

Let me know if that answers your question!
Oliver Schwarzwälder 0 Reputation points

2023-10-05T06:30:20.9133333+00:00
@SadiqhAhmed-MSFT Thank you for the fast reply.
The Vault was new defined without any existing data, it is a test environment in which we are facing the issues.
I already considered the Knowledge Base Articles and did Troubleshooting in doublecheck with existing working rsv and the only difference between them is that the CMK is enabled and in use
-> last steps i tried was manually set permissions between the vault to the machine

to be sure that access is granted and the agent can provide it's commands to the VSS within Windows 10 Pro

In addition the error message wich i get in Azure.
Oliver Schwarzwälder 0 Reputation points

2023-10-05T07:03:32.51+00:00

#Screenshot Screenshot 2023-10-05 at 09.01.59.png Azure/OS combined attached
SadiqhAhmed-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-05T13:43:54.7533333+00:00

@Oliver Schwarzwälder We will need to review the logs and troubleshoot the issue. Please open a support ticket for further investigation.
SadiqhAhmed-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-06T14:38:13.28+00:00

@Oliver Schwarzwälder Did you get a chance to contact Azure technical support team? Let us know if there are any blockers.
Oliver Schwarzwälder 0 Reputation points

2023-10-10T10:10:43.65+00:00

@SadiqhAhmed-MSFT , I contacted our MS-Distributor for Support, cause i couldn't open a support case directly.

Looking forward for the remote Support Session and it's outcome.

Thank you asking for further assistance, i keep you updated for the outcome and will post the resolution of the issue.
SadiqhAhmed-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-11T07:17:59.87+00:00

@Oliver Schwarzwälder Thanks for the update! :)
SadiqhAhmed-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-16T10:08:53.5333333+00:00

@Oliver Schwarzwälder Any progress on this?
Oliver Schwarzwälder 0 Reputation points

2023-10-16T14:35:32.6833333+00:00

@SadiqhAhmed-MSFT thank you for reply, at the moment we're in corresponding for changes besides the cmk vault, even the hypervisor scheduled task seems to fail with an path error for the encryption vault.

Keep you updated if the solution gets provided.
SadiqhAhmed-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-17T09:22:13.9+00:00

@Oliver Schwarzwälder Thanks for keeping us updated!
SadiqhAhmed-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-23T07:12:19.4333333+00:00

@Oliver Schwarzwälder Were you able to get a solution from Azure support team?

1 answer

Your answer

SadiqhAhmed-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-05T03:41:26.3233333+00:00

@Oliver Schwarzwälder Thank you for reaching out to us on Microsoft Q&A platform. Happy to assist you!

I understand that you are facing issues when using a CMK encrypted recovery services vault.

Encryption of backup data using customer-managed keys - Azure Backup | Microsoft Learn

Based on the move back and forth, going to guess this is relevant:

This feature allows you to encrypt new Recovery Services vaults only. Any vaults containing existing items registered or attempted to be registered to it aren't supported.

Let me know if that answers your question!
Oliver Schwarzwälder 0 Reputation points

2023-10-05T06:30:20.9133333+00:00

@SadiqhAhmed-MSFT Thank you for the fast reply.
The Vault was new defined without any existing data, it is a test environment in which we are facing the issues.
I already considered the Knowledge Base Articles and did Troubleshooting in doublecheck with existing working rsv and the only difference between them is that the CMK is enabled and in use
-> last steps i tried was manually set permissions between the vault to the machine

to be sure that access is granted and the agent can provide it's commands to the VSS within Windows 10 Pro

In addition the error message wich i get in Azure.
Oliver Schwarzwälder 0 Reputation points

2023-10-05T07:03:32.51+00:00

#Screenshot Screenshot 2023-10-05 at 09.01.59.png Azure/OS combined attached
SadiqhAhmed-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-05T13:43:54.7533333+00:00

@Oliver Schwarzwälder We will need to review the logs and troubleshoot the issue. Please open a support ticket for further investigation.
SadiqhAhmed-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-06T14:38:13.28+00:00

@Oliver Schwarzwälder Did you get a chance to contact Azure technical support team? Let us know if there are any blockers.
Oliver Schwarzwälder 0 Reputation points

2023-10-10T10:10:43.65+00:00

@SadiqhAhmed-MSFT , I contacted our MS-Distributor for Support, cause i couldn't open a support case directly.

Looking forward for the remote Support Session and it's outcome.

Thank you asking for further assistance, i keep you updated for the outcome and will post the resolution of the issue.
SadiqhAhmed-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-11T07:17:59.87+00:00

@Oliver Schwarzwälder Thanks for the update! :)
SadiqhAhmed-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-16T10:08:53.5333333+00:00

@Oliver Schwarzwälder Any progress on this?
Oliver Schwarzwälder 0 Reputation points

2023-10-16T14:35:32.6833333+00:00

@SadiqhAhmed-MSFT thank you for reply, at the moment we're in corresponding for changes besides the cmk vault, even the hypervisor scheduled task seems to fail with an path error for the encryption vault.

Keep you updated if the solution gets provided.
SadiqhAhmed-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-17T09:22:13.9+00:00

@Oliver Schwarzwälder Thanks for keeping us updated!
SadiqhAhmed-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-23T07:12:19.4333333+00:00

@Oliver Schwarzwälder Were you able to get a solution from Azure support team?

Answer 1

Oliver Schwarzwälder 0

@SadiqhAhmed-MSFT , finally we could face the root of the issue

-> it's the hardening combination of the CMK Key Vault with disable public access and the user assigned identity (also Access policies are not used -> RBAC is in use)

-> somehow there's a problem with access the key Vault within the permission set of the user assigned identity

when it's defined with system assigned and the recovery services vault gets the permission set for the key vault with RBAC it works fine

The request can be closed.
Thank you very much for stay in touch and interested in the resolution of the issue.

SadiqhAhmed-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-25T07:42:07.2366667+00:00

@Oliver Schwarzwälder Thanks for the update! Glad to hear it is resolved :)

Share via

VM and recovery services vault with CMK, snapshot for backup fails

1 answer

Your answer