Enable replication for VM failed in Azure sste recovery.

Shah, Darshan (Contractor) 26

Hello,

We are facing one issue while enabling replication for one database VM in Azure site recovery. These VMs have Always on Availability group configured and these VMs are behind the internal standard load balancer.

We have already configured below required outbound rules for NSG.

Still while enabling replication got below error.

Error Details

Installing Mobility Service and preparing target

Error ID
539

Error Message
The requested action couldn't be performed by the Replication Provider.

Possible causes
The Provider action failed. Check other errors for more information.

Recommendation
Resolve the issue and retry the operation.

First Seen At
1/17/2022, 4:10:50 PM

Error ID
151196

Error Message
Site recovery configuration failed.

Possible causes
Connection cannot be established to Office 365 authentication and identity IP4 endpoints.

Recommendation
Azure Site Recovery required access to Office 365 IPs ranges for authentication. 1. If you are using standard load balancer, then enable outbound access for login.microsoftonline.com. 2. If you are using Azure Network Security group (NSG) rules/firewall proxy to control outbound network connectivity on the VM, ensure you allow communication to the prerequisite URLs or service tags. Refer to https://aka.ms/a2a-troubleshoot-151196

Related links
https://aka.ms/a2a-troubleshoot-151196
First Seen At
1/17/2022, 4:10:50 PM

These are production database VMs so assigning public IP or changing the standard load balancer to Basic is not recommended.
Could anyone please guide on this ?

SadiqhAhmed-MSFT 37,686 Reputation points Microsoft Employee

2022-01-20T07:36:47.817+00:00

@Shah, Darshan (Contractor) Welcome to Microsoft Q&A community forum!

You have SQL always on cluster configured. If you are using shared disk for clustering, then it is unsupported. Also, we need to look from many other aspects. If you have followed the recommended action then IMO, this might require thorough investigation, maybe a support ticket will help.
Shah, Darshan (Contractor) 26 Reputation points

2022-01-20T07:39:51.537+00:00

Hello SadiqhAhmed,

No, in this production environment there is no shared disks are used in cluster.
SadiqhAhmed-MSFT 37,686 Reputation points Microsoft Employee

2022-01-20T07:44:01.99+00:00

The issue needs a further deeper dive technically. I recommend you to create a technical support ticket. The ticket enables you to work closely with the support engineers and get a quick resolution to your issue.

Here is the link - https://learn.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request to create support case.

Feel free to reply if you have any challenged contacting Azure technical support team.
Shah, Darshan (Contractor) 26 Reputation points

2022-01-25T13:13:07.6+00:00

Hello SadiqhAhmed,

After adding NAT Gateway , now we are able to enable replication. After synchronization process since long time replication status is "Waiting for first recovery point".

In Events - Last 72 hours(24) we found below messages since last 3 hours.

Recovery point generation failed.
App-consistent recovery point generation failed.

Do you have any hint ?

2 answers

SadiqhAhmed-MSFT 37,686 Reputation points Microsoft Employee

2022-01-27T10:43:13.71+00:00
The error - "no app-consistent recovery point in the last 'x' hours" when seen together with the continuous occurrence of the event- "app-consistent recovery tag generation failed" indicates that these errors are occurring due to failures while attempting to generate VSS snapshots.

On Windows servers, often times this can happen due to applications such as SQL or third party backup software attempting to generate VSS snapshots and failing/resulting in a clash with ASR's attempts to generate an application consistent snapshot.

You can find out if this is being caused due to a specific application/writer by running the following command from command prompt on the virtual machine on which this problem is being seen

"VSSadmin.exe list writers"

If the output of this command indicates failure for one particular writer, it is probably that application that is causing the VSS snapshots to fail. In such a case, most likely you'll be able to remediate this problem by restarting the service associated with the VSS writer that is failing.

If all the writers are in a healthy/stable state, try restarting the following services from services.msc

Volume Shadow Copy

Azure Site Recovery VSS Provider

Wait for a couple of hours once this is done to see if App-consistent snapshots are being generated successfully

Rebooting the VM is a another way to achieve what I've described here.

If you aren't interested in generating app-consistent recovery points for this virtual machine, you can change the app-consistent snapshot frequency in the replication policy to 0. Please not this will impact all virtual machines using this replication policy and ASR will stop attempting to generate app-consistent recovery points for virtual machines using this policy (ASR will then generate only crash consistent recovery points for these virtual machines)

Hope this helps!

If the response helped, do "Accept Answer" and up-vote it
Please sign in to rate this answer.
Shah, Darshan (Contractor) 26 Reputation points

2022-01-28T05:45:41.11+00:00

Hello SadiqhAhmed-MSFT,

As per your above comments I see that both Volume Shadow Copy and Azure Site Recovery VSS Provider services are stopped this VM. Should it in running mode ? I checked these services are also stopped on other VMs which are already replicated.
After 3-4 hours when this replication was not completed then I disabled the replication and then I can see below event in site recovery event.

Replication Health Errors Error ID 153018 Error Message
The data change rate (write bytes/sec) for Servername-disk-2 has exceeded ASR supported limits. This could be due to high I/O operations on disk caused by applications on the guest OS.
• Possible causes
The data change rate (observed for 60 mins) on the below disks of the virtual machine is more than the Azure Site Recovery supported limits for the target storage type. 1. Servername-disk-2, Churn rate : 20.37MBps, Observation start time : 01/25/2022 07:33:38 UTC, Churn rate limit : 20MBps

This disk is premium storage type P30. As per the churn rate limit it seems if it is more 20 MBs then replication will not work. What can we do in this ?

SadiqhAhmed-MSFT 37,686 Reputation points Microsoft Employee

2022-01-28T06:56:59.993+00:00

What is the workload being run on that VM?

Shah, Darshan (Contractor) 26 Reputation points

2022-01-28T07:03:27.803+00:00

This is database VM with SQL server 2016 with Always on availability group configured. Database is OLTP but not much busy all the time. Some scheduled jobs are running on this server daily. However no job which takes more than few seconds. (Not more than 1 minute).

This VM is also part of daily Azure VM backup. But when we configured replication this backup was already completed.

As per the Microsoft solution https://learn.microsoft.com/en-us/azure/site-recovery/azure-to-azure-troubleshoot-replication#solution, if load is temporary then replication will catch up automatically. But in my case after 2-3 hours it couldn't catch up and completed the replication.

SadiqhAhmed-MSFT 37,686 Reputation points Microsoft Employee

2022-01-28T09:24:53.157+00:00

Since its a SQL server I recommend following my previous post. Even after restarting the services, if replication fails, reboot the VM once.

Shah, Darshan (Contractor) 26 Reputation points

2022-01-28T09:36:11.587+00:00

Ok. Let me try by starting below two services.

Volume Shadow Copy
Azure Site Recovery VSS Provider services

I will update you on this. Thanks for your quick response.

Shah, Darshan (Contractor) 26 Reputation points

2022-01-28T10:28:31.157+00:00

SadiquAhmed,

One question. I have configured ASR for few Test VM also. On those servers below services are also not in running stats , still I could configure replication successfully. And those VMs are protected. So question is, should these two services always in running mode?

Volume Shadow Copy

Azure Site Recovery VSS Provider

SadiqhAhmed-MSFT 37,686 Reputation points Microsoft Employee

2022-01-28T11:17:50.387+00:00

Are those test VMs running SQL server?

Shah, Darshan (Contractor) 26 Reputation points

2022-01-28T11:29:45.603+00:00

Yes that VM is running with SQL server 2016. I also set up Azure backup for SQL VM on that VM. And it is also part of daily Azure VM backup.

169436-vss.jpg

SadiqhAhmed-MSFT 37,686 Reputation points Microsoft Employee

2022-01-28T15:08:02.103+00:00

Few points to be noted here:

Ensure these services are running:
Volume Shadow Copy
Azure Site Recovery VSS Provider services

Azure VM backup also perform app consistent backups so that might be causing this failure.
You may want to disabled Azure VM backup just to isolate the issue and see if replication happens successfully.

If nothing work, a reboot of the machine might be helpful.

Shah, Darshan (Contractor) 26 Reputation points

2022-01-31T07:54:50.253+00:00

Hello,

After running below command only one writer failed.

VSSadmin.exe list writers

Writer name: 'Cluster Shared Volume VSS Writer'
Writer Id: {1072ae1c-e5a7-4ea1-9e4a-6f7964656570}
Writer Instance Id: {d2eea877-8a09-4e1d-9048-4f0598881e1f}
State: [8] Failed
Last error: Timed out

Also observed that churn rate on data disk is constantly above 20 MBps after enabling the replication for that VM.

SadiqhAhmed-MSFT 37,686 Reputation points Microsoft Employee

2022-01-31T09:32:32.297+00:00

Not sure about churn rate. But if you are still blocked, I recommend contacting Azure technical support ticket for further investigation/assistance with this issue.

Shah, Darshan (Contractor) 26 Reputation points

2022-01-31T12:19:45.143+00:00

Hello SadiqhAhmed-MSFT

Ok. Will raise request for MS support.

Thanks for your help.

SadiqhAhmed-MSFT 37,686 Reputation points Microsoft Employee

2022-02-02T06:13:48.687+00:00

Any progress from support side?

Shah, Darshan (Contractor) 26 Reputation points

2022-02-02T07:13:40.287+00:00

Hello

We found that one trace file is running on this VM and its write rate is 4MBps. Currently we are analyzing what is this trace and will stop if it is not required. By this way we can reduce the disk write rate and make it in ASR limit. If this will not work then we will raise support ticket.
Sign in to comment
Shah, Darshan (Contractor) 26 Reputation points

2022-02-04T08:12:37.993+00:00

To resolve below error ,I have added NAT Gateway , now we are able to enable replication.
Error ID
539

Error Message
The requested action couldn't be performed by the Replication Provider.
Please sign in to rate this answer.
SadiqhAhmed-MSFT 37,686 Reputation points Microsoft Employee

2022-02-04T10:34:03.977+00:00

Thanks for the update. Glad to know you were able to resolve this issue. So, adding NAT Gateway fixed the issue and replication started as expected.
Sign in to comment