Having problem backing up a file disk using DPM to the cloud.

Question

Having problem backing up a file disk using DPM to the cloud.

Adam Staub 26

Try to backup some flat files on a volume to the cloud using MS DPM. Other cloud backups are working fine. However, This one and a couple of others are struggling. Looking thru the logs I'm seeing this, as shown below. What is interesting to me is the log talks about ErrorCode:CloudAsyncWorkLimitReached is there some I can do on my end to adjust this or is this a MS thing? It looks like it hits this cloudAsyncWorklimitReached deal then does a retry, looping over and over. I've run a consistency check on the Volume. and also moved the volume to a different protection group. Same error. Does anybody have an Idea?

17D0 2518 02/26 18:31:37.698 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING FMBlock: Executing retry policy | Params: {CurrentAttempt = 5}{MaxAttempts = 125}{BackOffIntervalInSec = 18}{OperationCode = WcfProxyAsyncCall}{Retry policy id = 8}
17D0 0CEC 02/26 18:31:39.011 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 0CEC 02/26 18:31:41.027 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 0CEC 02/26 18:31:43.043 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 0CEC 02/26 18:31:45.059 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 0CEC 02/26 18:31:47.074 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 0CEC 02/26 18:31:49.090 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 2518 02/26 18:31:50.751 79 WcfClient.cs(960) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL Executing web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.FileCatalog.Interface.IFileCatalogExternalChannel}{Description = UpdateROArgs}{TargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}
17D0 2518 02/26 18:31:50.751 69 CBEngineWcfClientHelper.cs(396) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL Setting RequestId header for outgoing request | Params: {RequestId = 9fb0ab40-b186-47d2-90c2-7dfe0e9af1d4}{WorkitemId = 87b20db0-7e29-49d4-95dd-9953fffee9ba}{TaskId = 41f5d7db-7b3e-4564-9c04-522ae6448e4d}{AgentVersion = 2.0.9197.0}
17D0 0CEC 02/26 18:31:51.094 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING Unable to make web service call | Params: {Exception:  = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING  (Fault Detail is equal to ErrorCode = CloudAsyncWorkLimitReached, DetailedErrorCode = 0, DetailedErrorSource = None/None, Message = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ).}
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyCall}{OperationStartTime = 2/26/2021 6:31:50 PM}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ]
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING {InitialTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{Name = BasicHttpBinding_IFileCatalogExternal}{WorkitemId = 87b20db0-7e29-49d4-95dd-9953fffee9ba}{CurrentTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{RequestId = 0000000002_9fb0ab40-b186-47d2-90c2-7dfe0e9af1d4_41f5d7db-7b3e-4564-9c04-522ae6448e4d}{IdentityType = Container}{Identity = 50aebc141d99421416176dd13c4715a0f709e7d2}
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING }
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL FMBlock: No retry policy specified | Params: {ErrorCode = CloudAsyncWorkLimitReached}
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL FMBlock: Mapping source error to target error | Params: {OperationCode = WcfProxyCall}{SourceError = CloudAsyncWorkLimitReached}{TargetError = CloudAsyncWorkLimitReached}
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyAsyncCall}{OperationStartTime = 2/26/2021 6:30:56 PM}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ]
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING {InitialTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{Name = BasicHttpBinding_IFileCatalogExternal}{WorkitemId = 87b20db0-7e29-49d4-95dd-9953fffee9ba}{CurrentTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{RequestId = 0000000002_9fb0ab40-b186-47d2-90c2-7dfe0e9af1d4_41f5d7db-7b3e-4564-9c04-522ae6448e4d}{IdentityType = Container}{Identity = 50aebc141d99421416176dd13c4715a0f709e7d2}
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING }

17D0 0CEC 02/26 18:32:13.196 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 0CEC 02/26 18:32:15.212 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 0CEC 02/26 18:32:17.215 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 0CEC 02/26 18:32:19.231 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 2518 02/26 18:32:19.861 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING Unable to make web service call | Params: {Exception:  = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
17D0 2518 02/26 18:32:19.861 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING  (Fault Detail is equal to ErrorCode = CloudAsyncWorkLimitReached, DetailedErrorCode = 0, DetailedErrorSource = None/None, Message = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
17D0 2518 02/26 18:32:19.861 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ).}
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyCall}{OperationStartTime = 2/26/2021 6:32:09 PM}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ]
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING {InitialTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{Name = BasicHttpBinding_IFileCatalogExternal}{WorkitemId = 5bbe5f65-f6a6-4570-a5de-f12605d09947}{CurrentTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{RequestId = 0000000002_b2161111-b2e7-457d-8eee-f30a7aa6939c_41f5d7db-7b3e-4564-9c04-522ae6448e4d}{IdentityType = Container}{Identity = 50aebc141d99421416176dd13c4715a0f709e7d2}
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING }
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL FMBlock: No retry policy specified | Params: {ErrorCode = CloudAsyncWorkLimitReached}
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL FMBlock: Mapping source error to target error | Params: {OperationCode = WcfProxyCall}{SourceError = CloudAsyncWorkLimitReached}{TargetError = CloudAsyncWorkLimitReached}
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyAsyncCall}{OperationStartTime = 2/26/2021 6:30:56 PM}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ]
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING {InitialTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{Name = BasicHttpBinding_IFileCatalogExternal}{WorkitemId = 5bbe5f65-f6a6-4570-a5de-f12605d09947}{CurrentTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{RequestId = 0000000002_b2161111-b2e7-457d-8eee-f30a7aa6939c_41f5d7db-7b3e-4564-9c04-522ae6448e4d}{IdentityType = Container}{Identity = 50aebc141d99421416176dd13c4715a0f709e7d2}
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING }

Karen Payne MVP 35,586 Reputation points Volunteer Moderator

2021-02-26T18:17:05.717+00:00

Hello.

Please format your post so we can read it rather than have everything together.
James Williams 21 Reputation points

2021-02-27T15:41:41.847+00:00

Getting the same error with a number of recovery points on my MABS server. The three recovery points are all file-based.

I'm also in a different region
https://pod01-fc1.ukw.backup.windowsazure.com/FileCatalogExternalService.svc
James Williams 21 Reputation points

2021-03-01T11:11:46.47+00:00

I've now had this reported it is an Azure issue and MS are looking into it. Not sure why they haven't updated the status page though
Adam Staub 26 Reputation points

2021-03-01T17:25:03.963+00:00

James:
How did you find out that this is a MS deal?
SadiqhAhmed-MSFT 49,326 Reputation points Microsoft Employee Moderator

2021-03-01T17:30:56.603+00:00

@Adam Staub James reached out to Azure technical support team and the issue is being investigated.
Adam Staub 26 Reputation points

2021-03-01T17:34:55.627+00:00

So, MS recognizes that there is an issue? Or are they investigating if there is an issue? SQL backups work fine. Just having problems with flat file backups.

I'm getting the exact same type of message as he is. Will results of investigation be post here?
James Williams 21 Reputation points

2021-03-01T18:26:27.75+00:00

They recognize it is an issue. I've asked multiple times why it hasn't been flagged on the health status portal. However, the support agent I'm working with isn't being very responsive and wants me to wait two days!

Hopefully Sadiq can push this forward.
Adam Staub 26 Reputation points

2021-03-01T19:36:29.423+00:00

Dang for me this has been going on since like last Thursday maybe? I can't remember. Not having backups is NOT good! :)
pissteufel 6 Reputation points

2021-03-02T08:03:14.67+00:00

Hi,

a customer of mine is also having trouble with file-based online backups performed by Azure Backup Server through Azure Recovery Services Agent. Some backup jobs don' t finish. In CBengineCurr.errlog are a lot of entries with error message "ErrorCode = CloudAsyncWorkLimitReached, DetailedErrorCode = 0, DetailedErrorSource = None/None, Message = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]"

The Azure Backup endpoint mentioned in the error message blocks is "InitialTargetEndpoint = https://pod01-fc1.we.backup.windowsazure.com/FileCatalogExternalService.svc"

So, I am also interested in a solution.

Regards,
James Williams 21 Reputation points

2021-03-02T08:17:32.32+00:00

MS have come back and said:

We have made some changes on backend and backups which will start today, will continue to run normally.
Let the current backups finish, after that the next ones would run fine.

I'm still getting the error in the logs and the jobs are still running very slowly so am skeptical. Have asked if I should cancel and restart jobs
Southall, Mark 1 Reputation point

2021-03-02T11:43:17.7+00:00
We've also been receiving this error, and noticing backups taking progressively longer since it started.

We've kicked off an online recovery point job around an hour and 20 minutes ago, and the data transfer counter hasn't updated for 40 minutes.

We're still getting the same error within the CBEngineCurr.errlog file, with

Unable to make web service call | Params: {Exception: = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]

InitialTargetEndpoint = https://pod01-fc1.uks.backup.windowsazure.com/FileCatalogExternalService.svc
James Williams 21 Reputation points

2021-03-02T11:44:31.117+00:00

Cancelling and restarting jobs hasn't fixed :(
pissteufel 6 Reputation points

2021-03-02T13:21:45.15+00:00

We also cancelled the remaining online backup jobs and in addition restarted Azure Backup Server. After that we run a new online backup job, but still the error is logged multiple times.
Adam Staub 26 Reputation points

2021-03-02T15:44:25.267+00:00

I restarted cbengine.exe and ran a backup job. Failing in the same place as before. with the same errors. Either the change didn't work or the change hasn't reach the Central US region yet. James do you have a case number that we could refer to or anything like that? I'm glad it is biting more than just one of us that might push MS to fix the issue quickly.
James Williams 21 Reputation points

2021-03-02T17:54:16.583+00:00

Still having issues. I'll be honest I've been incredibly disappointed with the Azure backup team on this case. Usually, they are pretty good but this has been shocking. They really don't seem to consider backups a priority.
Steve 1 Reputation point

2021-03-02T19:21:52.987+00:00

Hi, I'm in the UK and having the exact same problem, just adding my comment so MS realise it's not an isolated issue.
Adam Staub 26 Reputation points

2021-03-02T19:40:10.093+00:00

Just checked again. Things aren't working. (: . I wonder if the moderator of this forum could reach out for an update, @James Williams .

@SadiqhAhmed-MSFT Could you get us an update on this?
James Williams 21 Reputation points

2021-03-02T20:01:08.873+00:00

Latest on my case is they've pushed out a fix but might take some time to replicate to all regions
Adam Staub 26 Reputation points

2021-03-02T20:58:22.357+00:00

Awesome! Hopefully it works. Thanks, for keeping all of us in the loop.
Dave A 1 Reputation point

2021-03-03T05:58:21.483+00:00

I'm also experiencing this issue....

Just killed off the running online backups, rebooted and kicked a job off.

Not looking good. 17mb in 90 mins
pissteufel 6 Reputation points

2021-03-03T11:14:50.927+00:00

Status info: According Azure region "west europe" error is still happening.
Adam Staub 26 Reputation points

2021-03-03T15:15:18.98+00:00

Not looking very good here either. The backup got further along but still hung. Checked the logs and am getting the same error. Is there anybody in the forum who could tell us if the fix rolled world-wide last night?
Steve 1 Reputation point

2021-03-03T15:24:35.763+00:00

Not working here either still (UK South). Though it did manage a whole 1GB before hanging which is an improvement.

Nearly a week now without a backup to Azure....

1A94 14D0 03/03 15:22:32.293 79 WcfClient.cs(998) B1CA532D-09ED-4719-91C0-07088A3D333E WARNING Unable to make web service call | Params: {Exception: = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
Adam Staub 26 Reputation points

2021-03-03T15:33:30.097+00:00

Exact same error as before.
Adam Staub 26 Reputation points

2021-03-03T16:17:28.373+00:00

Does the moderator have the ability to find out more? Would be nice if we got an update? @SadiqhAhmed-MSFT Could you provide use an update? @James Williams Did you find out anything more?
jham6509 1 Reputation point

2021-03-03T17:18:59.813+00:00

Having the same issue in the US region. Been happing since the beginning of this week. SQL backups work fine, file backups are not progressing.
jham6509 1 Reputation point

2021-03-03T17:37:23.9+00:00

Where did you get the Azure backup agent version 2.0.9205.0 from and is that the latest?

The latest I'm seeing is 9.0.9200.0 from here https://learn.microsoft.com/en-us/azure/backup/install-mars-agent#download-the-mars-agent.
James Williams 21 Reputation points

2021-03-03T22:27:51.797+00:00

Sadly MS have not progressed this any further.

We have just got another update that product group team has reverted to the older build and has already started the deployment process. They are replicating the changes to azure regions and I checked job history from your environment for today and a lot of jobs are completing in time.

Like all of you, my file backups still aren't working.
Dave A 1 Reputation point

2021-03-04T06:36:39.797+00:00

14:30 in Australia, still an issue for file based backup.

Does someone have a MS incident number I can quote for this?
James Williams 21 Reputation points

2021-03-04T08:21:06.443+00:00

My case number is 2103010050000340
Steve 1 Reputation point

2021-03-04T09:08:37.997+00:00

Don't want to speak too soon, but one of my online recovery points (which I was about to cancel as it had only uploaded 200MB in 9 hours) has suddenly kicked back into life. It's uploaded another 1.5GB in the past 30 minutes with no errors in the log (at the moment). Might be fixed.. (UK South)
Martin-1160 1 Reputation point

2021-03-04T12:32:02.787+00:00

Hitting UKS here... still seeing issues. getting byte counts but jobs never complete (kicked off 2 hours ago).
The longest i have left a job is 36 hours (never completed).
Our Exchange, SQL, VMWare backups all function fine, just file based as everyone else is saying.

CurrentTargetEndpoint = https://pod01-fc1.uks.backup.windowsazure.com/FileCatalogExternalService.svc

281C 27C4 03/04 12:28:41.911 79 WcfClient.cs(694) 11367A21-500B-4B24-941D-A7521D9FE283 WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyAsyncCall}{OperationStartTime = 04/03/2021 12:28:31}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
Virtuworks 6 Reputation points

2021-03-05T01:31:27.1+00:00

Anyone have any further information on this? We are also seeing this issue in EAST US
Dave A 1 Reputation point

2021-03-05T06:54:12.183+00:00

Well, I've just got through a full file backup for the first time in a week maybe?

Will test out another server now.
Dave A 1 Reputation point

2021-03-05T07:28:36.053+00:00

Second server backup complete 4gb data transferred

No changes were made on my end for this to start working.
Steve 1 Reputation point

2021-03-05T07:56:35.087+00:00

My job from yesterday has failed, it uploaded about 15GB in a couple of hours and has hung, it's still running now (34 hours) but has not uploaded anything for about 24 hours. I will cancel it and try again, but the problem seems to be still here.

Another job has been running for 9 hours and has uploaded 3GB before also hanging, will be cancelling that one in a minute too.
Adam Staub 26 Reputation points

2021-03-05T14:55:18.24+00:00

What region are you in @Dave A
Adam Staub 26 Reputation points

2021-03-05T14:55:45.69+00:00

I'm usa west. Still no go here............
Virtuworks 6 Reputation points

2021-03-05T18:07:09.693+00:00

Guys I spoke with the Azure Backup Product manager today. As others say there is a fix being pushed which should hit all Regions by Sunday Night. East coast is scheduled for 10pm March 6th. Nothing can be done but wait for this patch to apply.
James Williams 21 Reputation points

2021-03-05T19:15:59.593+00:00

Looking good in UK-WEST!
pissteufel 6 Reputation points

2021-03-06T10:28:13.09+00:00

Let's hope for the best! Our customers get worried because no file based online backup could be finished for a while because of these error.
Steve 1 Reputation point

2021-03-06T13:23:28.22+00:00

It was working, got 2 backups complete and now it's hung and errored out at 16GB uploaded on my 3rd backup
Steve 1 Reputation point

2021-03-06T13:28:25.9+00:00

Slightly different error now though -

1A94 05B8 03/06 13:25:12.566 79 WcfClient.cs(998) 3CB84549-500E-4D8D-9970-BA3C1E42353D WARNING Unable to make web service call | Params: {Exception: = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkSubmitted, DetailedCode:0, Source:None, Message:]
1A94 05B8 03/06 13:25:12.566 79 WcfClient.cs(998) 3CB84549-500E-4D8D-9970-BA3C1E42353D WARNING (Fault Detail is equal to ErrorCode = CloudAsyncWorkSubmitted, DetailedErrorCode = 0, DetailedErrorSource = None/None, Message = FMException: [ErrorCode:CloudAsyncWorkSubmitted, DetailedCode:0, Source:None, Message:]
1A94 05B8 03/06 13:25:12.566 79 WcfClient.cs(998) 3CB84549-500E-4D8D-9970-BA3C1E42353D WARNING ).}
1A94 05B8 03/06 13:25:12.566 79 WcfClient.cs(724) 3CB84549-500E-4D8D-9970-BA3C1E42353D NORMAL Request submitted for async processing
Dave A 1 Reputation point

2021-03-08T00:01:27.707+00:00

Sorry for the delay in response... no work on the weekend for me :)

I'm in Australia.

Over the weekend, all but one of my backups were successful.

The one that didn't finish had been running for 55 hours, I have just canned that and restarted.
SadiqhAhmed-MSFT 49,326 Reputation points Microsoft Employee Moderator

2021-03-08T07:43:13.86+00:00

Hello All,

Sorry for the inconvenience this must have caused!

This is a known issue and has been mitigated. The fix has been deployed in all the regions. The issue should get resolved by EOD.

Appreciate your patience in this matter.

Thanks!
Martin-1160 1 Reputation point

2021-03-08T14:01:02.407+00:00

confirmed UKSouth is now functioning normally
James Williams 21 Reputation points

2021-03-08T20:51:54.023+00:00

Mine are all running and have completed. Although some jobs are running much slower than normal. Anyone else finding this?
Dave A 1 Reputation point

2021-03-09T06:37:49.63+00:00

Not noticing any real slowness...

Still having trouble with one particular server but it may be a local issue.
pissteufel 6 Reputation points

2021-03-09T09:18:15.15+00:00

I am also struggeling with one particular volume of a bigger file server. Hopefully the new downloadable MARS version 2.0.9207.0 can help.
pissteufel 6 Reputation points

2021-03-10T15:39:45.92+00:00

@SadiqhAhmed-MSFT :
Still getting the following error in Azure region "west europe":
"WARNING Unable to make web service call | Params: {Exception: = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkSubmitted, DetailedCode:0, Source:None, Message:]

MARS agent version is 2.0.9207.0

Any hint?
Dave A 1 Reputation point

2021-03-11T01:06:38.533+00:00

Yeah I still have issue with one largish file server. I've left the job to run and it's up to 42 hours now, still transferring very slowly.

All other servers seem to be fine.

Has your situation changed?
pissteufel 6 Reputation points

2021-03-11T11:25:47.677+00:00

Nope, I have three jobs for multi terabyte volumes which run endless and transferring only few megabytes. According to cbengine log the jobs are running in unoptimized mode, which may explain the long job run-time. In addition I experienced a massive slow down in my disk-based jobs when running the remaining three jobs in parallel. It seems the online jobs and the correspond disk-based jobs block each other. As soon as I cancel the online jobs the disk-based jobs run fine. Because of the unoptimized mode the remaining online jobs don't finish in time and the disk-based jobs already start. My next workaround is to exclude the disk-based jobs on weekend to finish the remaining online jobs.
James Williams 21 Reputation points

2021-03-16T11:57:23.21+00:00

We seem all ok now
pissteufel 6 Reputation points

2021-03-16T21:32:00.967+00:00

Yes, we also. One job ran more then 60 hours but completed successfully.
jham6509 1 Reputation point

2021-03-16T21:38:02.28+00:00

Did you notice the job takes a lot longer than normal? Even though I have a few that are completing now, completion time seems to have increased by 5 times.

1 answer

Your answer

Karen Payne MVP 35,586 Reputation points Volunteer Moderator

2021-02-26T18:17:05.717+00:00

Hello.

Please format your post so we can read it rather than have everything together.
James Williams 21 Reputation points

2021-02-27T15:41:41.847+00:00

Getting the same error with a number of recovery points on my MABS server. The three recovery points are all file-based.

I'm also in a different region
https://pod01-fc1.ukw.backup.windowsazure.com/FileCatalogExternalService.svc
James Williams 21 Reputation points

2021-03-01T11:11:46.47+00:00

I've now had this reported it is an Azure issue and MS are looking into it. Not sure why they haven't updated the status page though
Adam Staub 26 Reputation points

2021-03-01T17:25:03.963+00:00

James:
How did you find out that this is a MS deal?
SadiqhAhmed-MSFT 49,326 Reputation points Microsoft Employee Moderator

2021-03-01T17:30:56.603+00:00

@Adam Staub James reached out to Azure technical support team and the issue is being investigated.
Adam Staub 26 Reputation points

2021-03-01T17:34:55.627+00:00

So, MS recognizes that there is an issue? Or are they investigating if there is an issue? SQL backups work fine. Just having problems with flat file backups.

I'm getting the exact same type of message as he is. Will results of investigation be post here?
James Williams 21 Reputation points

2021-03-01T18:26:27.75+00:00

They recognize it is an issue. I've asked multiple times why it hasn't been flagged on the health status portal. However, the support agent I'm working with isn't being very responsive and wants me to wait two days!

Hopefully Sadiq can push this forward.
Adam Staub 26 Reputation points

2021-03-01T19:36:29.423+00:00

Dang for me this has been going on since like last Thursday maybe? I can't remember. Not having backups is NOT good! :)
pissteufel 6 Reputation points

2021-03-02T08:03:14.67+00:00

Hi,

a customer of mine is also having trouble with file-based online backups performed by Azure Backup Server through Azure Recovery Services Agent. Some backup jobs don' t finish. In CBengineCurr.errlog are a lot of entries with error message "ErrorCode = CloudAsyncWorkLimitReached, DetailedErrorCode = 0, DetailedErrorSource = None/None, Message = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]"

The Azure Backup endpoint mentioned in the error message blocks is "InitialTargetEndpoint = https://pod01-fc1.we.backup.windowsazure.com/FileCatalogExternalService.svc"

So, I am also interested in a solution.

Regards,
James Williams 21 Reputation points

2021-03-02T08:17:32.32+00:00

MS have come back and said:

We have made some changes on backend and backups which will start today, will continue to run normally.
Let the current backups finish, after that the next ones would run fine.

I'm still getting the error in the logs and the jobs are still running very slowly so am skeptical. Have asked if I should cancel and restart jobs
Southall, Mark 1 Reputation point

2021-03-02T11:43:17.7+00:00

We've also been receiving this error, and noticing backups taking progressively longer since it started.

We've kicked off an online recovery point job around an hour and 20 minutes ago, and the data transfer counter hasn't updated for 40 minutes.

We're still getting the same error within the CBEngineCurr.errlog file, with

Unable to make web service call | Params: {Exception: = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]

InitialTargetEndpoint = https://pod01-fc1.uks.backup.windowsazure.com/FileCatalogExternalService.svc
James Williams 21 Reputation points

2021-03-02T11:44:31.117+00:00

Cancelling and restarting jobs hasn't fixed :(
pissteufel 6 Reputation points

2021-03-02T13:21:45.15+00:00

We also cancelled the remaining online backup jobs and in addition restarted Azure Backup Server. After that we run a new online backup job, but still the error is logged multiple times.
Adam Staub 26 Reputation points

2021-03-02T15:44:25.267+00:00

I restarted cbengine.exe and ran a backup job. Failing in the same place as before. with the same errors. Either the change didn't work or the change hasn't reach the Central US region yet. James do you have a case number that we could refer to or anything like that? I'm glad it is biting more than just one of us that might push MS to fix the issue quickly.
James Williams 21 Reputation points

2021-03-02T17:54:16.583+00:00

Still having issues. I'll be honest I've been incredibly disappointed with the Azure backup team on this case. Usually, they are pretty good but this has been shocking. They really don't seem to consider backups a priority.
Steve 1 Reputation point

2021-03-02T19:21:52.987+00:00

Hi, I'm in the UK and having the exact same problem, just adding my comment so MS realise it's not an isolated issue.
Adam Staub 26 Reputation points

2021-03-02T19:40:10.093+00:00

Just checked again. Things aren't working. (: . I wonder if the moderator of this forum could reach out for an update, @James Williams .

@SadiqhAhmed-MSFT Could you get us an update on this?
James Williams 21 Reputation points

2021-03-02T20:01:08.873+00:00

Latest on my case is they've pushed out a fix but might take some time to replicate to all regions
Adam Staub 26 Reputation points

2021-03-02T20:58:22.357+00:00

Awesome! Hopefully it works. Thanks, for keeping all of us in the loop.
Dave A 1 Reputation point

2021-03-03T05:58:21.483+00:00

I'm also experiencing this issue....

Just killed off the running online backups, rebooted and kicked a job off.

Not looking good. 17mb in 90 mins
pissteufel 6 Reputation points

2021-03-03T11:14:50.927+00:00

Status info: According Azure region "west europe" error is still happening.
Adam Staub 26 Reputation points

2021-03-03T15:15:18.98+00:00

Not looking very good here either. The backup got further along but still hung. Checked the logs and am getting the same error. Is there anybody in the forum who could tell us if the fix rolled world-wide last night?
Steve 1 Reputation point

2021-03-03T15:24:35.763+00:00

Not working here either still (UK South). Though it did manage a whole 1GB before hanging which is an improvement.

Nearly a week now without a backup to Azure....

1A94 14D0 03/03 15:22:32.293 79 WcfClient.cs(998) B1CA532D-09ED-4719-91C0-07088A3D333E WARNING Unable to make web service call | Params: {Exception: = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
Adam Staub 26 Reputation points

2021-03-03T15:33:30.097+00:00

Exact same error as before.
Adam Staub 26 Reputation points

2021-03-03T16:17:28.373+00:00

Does the moderator have the ability to find out more? Would be nice if we got an update? @SadiqhAhmed-MSFT Could you provide use an update? @James Williams Did you find out anything more?
jham6509 1 Reputation point

2021-03-03T17:18:59.813+00:00

Having the same issue in the US region. Been happing since the beginning of this week. SQL backups work fine, file backups are not progressing.
jham6509 1 Reputation point

2021-03-03T17:37:23.9+00:00

Where did you get the Azure backup agent version 2.0.9205.0 from and is that the latest?

The latest I'm seeing is 9.0.9200.0 from here https://learn.microsoft.com/en-us/azure/backup/install-mars-agent#download-the-mars-agent.
James Williams 21 Reputation points

2021-03-03T22:27:51.797+00:00

Sadly MS have not progressed this any further.

We have just got another update that product group team has reverted to the older build and has already started the deployment process. They are replicating the changes to azure regions and I checked job history from your environment for today and a lot of jobs are completing in time.

Like all of you, my file backups still aren't working.
Dave A 1 Reputation point

2021-03-04T06:36:39.797+00:00

14:30 in Australia, still an issue for file based backup.

Does someone have a MS incident number I can quote for this?
James Williams 21 Reputation points

2021-03-04T08:21:06.443+00:00

My case number is 2103010050000340
Steve 1 Reputation point

2021-03-04T09:08:37.997+00:00

Don't want to speak too soon, but one of my online recovery points (which I was about to cancel as it had only uploaded 200MB in 9 hours) has suddenly kicked back into life. It's uploaded another 1.5GB in the past 30 minutes with no errors in the log (at the moment). Might be fixed.. (UK South)
Martin-1160 1 Reputation point

2021-03-04T12:32:02.787+00:00

Hitting UKS here... still seeing issues. getting byte counts but jobs never complete (kicked off 2 hours ago).
The longest i have left a job is 36 hours (never completed).
Our Exchange, SQL, VMWare backups all function fine, just file based as everyone else is saying.

CurrentTargetEndpoint = https://pod01-fc1.uks.backup.windowsazure.com/FileCatalogExternalService.svc

281C 27C4 03/04 12:28:41.911 79 WcfClient.cs(694) 11367A21-500B-4B24-941D-A7521D9FE283 WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyAsyncCall}{OperationStartTime = 04/03/2021 12:28:31}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
Virtuworks 6 Reputation points

2021-03-05T01:31:27.1+00:00

Anyone have any further information on this? We are also seeing this issue in EAST US
Dave A 1 Reputation point

2021-03-05T06:54:12.183+00:00

Well, I've just got through a full file backup for the first time in a week maybe?

Will test out another server now.
Dave A 1 Reputation point

2021-03-05T07:28:36.053+00:00

Second server backup complete 4gb data transferred

No changes were made on my end for this to start working.
Steve 1 Reputation point

2021-03-05T07:56:35.087+00:00

My job from yesterday has failed, it uploaded about 15GB in a couple of hours and has hung, it's still running now (34 hours) but has not uploaded anything for about 24 hours. I will cancel it and try again, but the problem seems to be still here.

Another job has been running for 9 hours and has uploaded 3GB before also hanging, will be cancelling that one in a minute too.
Adam Staub 26 Reputation points

2021-03-05T14:55:18.24+00:00

What region are you in @Dave A
Adam Staub 26 Reputation points

2021-03-05T14:55:45.69+00:00

I'm usa west. Still no go here............
Virtuworks 6 Reputation points

2021-03-05T18:07:09.693+00:00

Guys I spoke with the Azure Backup Product manager today. As others say there is a fix being pushed which should hit all Regions by Sunday Night. East coast is scheduled for 10pm March 6th. Nothing can be done but wait for this patch to apply.
James Williams 21 Reputation points

2021-03-05T19:15:59.593+00:00

Looking good in UK-WEST!
pissteufel 6 Reputation points

2021-03-06T10:28:13.09+00:00

Let's hope for the best! Our customers get worried because no file based online backup could be finished for a while because of these error.
Steve 1 Reputation point

2021-03-06T13:23:28.22+00:00

It was working, got 2 backups complete and now it's hung and errored out at 16GB uploaded on my 3rd backup
Steve 1 Reputation point

2021-03-06T13:28:25.9+00:00

Slightly different error now though -

1A94 05B8 03/06 13:25:12.566 79 WcfClient.cs(998) 3CB84549-500E-4D8D-9970-BA3C1E42353D WARNING Unable to make web service call | Params: {Exception: = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkSubmitted, DetailedCode:0, Source:None, Message:]
1A94 05B8 03/06 13:25:12.566 79 WcfClient.cs(998) 3CB84549-500E-4D8D-9970-BA3C1E42353D WARNING (Fault Detail is equal to ErrorCode = CloudAsyncWorkSubmitted, DetailedErrorCode = 0, DetailedErrorSource = None/None, Message = FMException: [ErrorCode:CloudAsyncWorkSubmitted, DetailedCode:0, Source:None, Message:]
1A94 05B8 03/06 13:25:12.566 79 WcfClient.cs(998) 3CB84549-500E-4D8D-9970-BA3C1E42353D WARNING ).}
1A94 05B8 03/06 13:25:12.566 79 WcfClient.cs(724) 3CB84549-500E-4D8D-9970-BA3C1E42353D NORMAL Request submitted for async processing
Dave A 1 Reputation point

2021-03-08T00:01:27.707+00:00

Sorry for the delay in response... no work on the weekend for me :)

I'm in Australia.

Over the weekend, all but one of my backups were successful.

The one that didn't finish had been running for 55 hours, I have just canned that and restarted.
SadiqhAhmed-MSFT 49,326 Reputation points Microsoft Employee Moderator

2021-03-08T07:43:13.86+00:00

Hello All,

Sorry for the inconvenience this must have caused!

This is a known issue and has been mitigated. The fix has been deployed in all the regions. The issue should get resolved by EOD.

Appreciate your patience in this matter.

Thanks!
Martin-1160 1 Reputation point

2021-03-08T14:01:02.407+00:00

confirmed UKSouth is now functioning normally
James Williams 21 Reputation points

2021-03-08T20:51:54.023+00:00

Mine are all running and have completed. Although some jobs are running much slower than normal. Anyone else finding this?
Dave A 1 Reputation point

2021-03-09T06:37:49.63+00:00

Not noticing any real slowness...

Still having trouble with one particular server but it may be a local issue.
pissteufel 6 Reputation points

2021-03-09T09:18:15.15+00:00

I am also struggeling with one particular volume of a bigger file server. Hopefully the new downloadable MARS version 2.0.9207.0 can help.
pissteufel 6 Reputation points

2021-03-10T15:39:45.92+00:00

@SadiqhAhmed-MSFT :
Still getting the following error in Azure region "west europe":
"WARNING Unable to make web service call | Params: {Exception: = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkSubmitted, DetailedCode:0, Source:None, Message:]

MARS agent version is 2.0.9207.0

Any hint?
Dave A 1 Reputation point

2021-03-11T01:06:38.533+00:00

Yeah I still have issue with one largish file server. I've left the job to run and it's up to 42 hours now, still transferring very slowly.

All other servers seem to be fine.

Has your situation changed?
pissteufel 6 Reputation points

2021-03-11T11:25:47.677+00:00

Nope, I have three jobs for multi terabyte volumes which run endless and transferring only few megabytes. According to cbengine log the jobs are running in unoptimized mode, which may explain the long job run-time. In addition I experienced a massive slow down in my disk-based jobs when running the remaining three jobs in parallel. It seems the online jobs and the correspond disk-based jobs block each other. As soon as I cancel the online jobs the disk-based jobs run fine. Because of the unoptimized mode the remaining online jobs don't finish in time and the disk-based jobs already start. My next workaround is to exclude the disk-based jobs on weekend to finish the remaining online jobs.
James Williams 21 Reputation points

2021-03-16T11:57:23.21+00:00

We seem all ok now
pissteufel 6 Reputation points

2021-03-16T21:32:00.967+00:00

Yes, we also. One job ran more then 60 hours but completed successfully.
jham6509 1 Reputation point

2021-03-16T21:38:02.28+00:00

Did you notice the job takes a lot longer than normal? Even though I have a few that are completing now, completion time seems to have increased by 5 times.

Answer 1

1E2C    1C9C    02/27   15:57:55.451    79  WcfClient.cs(830)       E35FFF34-DF88-4333-A731-0305000562DF        NORMAL  GetStatus Arguments | Params: {lasttimeClientReceivedNewHeartBeat = 27/02/2021 15:57:38}{lastNewHeatbeat = 27/02/2021 15:57:38}{statusPollTime = 27/02/2021 15:57:55}
1E2C    1C9C    02/27   15:57:55.451    79  WcfClient.cs(960)       E35FFF34-DF88-4333-A731-0305000562DF        NORMAL  Executing web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.Protection.Interface.IProtectionServiceChannel}{Description = GetStatus}{TargetEndpoint = https://pod01-prot1.ukw.backup.windowsazure.com/ProtectionService.svc}
1E2C    1C9C    02/27   15:57:55.451    69  CBEngineWcfClientHelper.cs(396)     E35FFF34-DF88-4333-A731-0305000562DF        NORMAL  Setting RequestId header for outgoing request | Params: {RequestId = 3f1440e2-6d52-4fc0-9915-ce50cae9550c}{WorkitemId = acb2a9ea-e8b5-4368-8bf9-2f85f0b67ae2}{TaskId = e35fff34-df88-4333-a731-0305000562df}{AgentVersion = 2.0.9205.0}
1E2C    1C9C    02/27   15:57:55.498    79  WcfClient.cs(981)       E35FFF34-DF88-4333-A731-0305000562DF        NORMAL  Finished web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.Protection.Interface.IProtectionServiceChannel}{Description = GetStatus}{TargetEndpoint = https://pod01-prot1.ukw.backup.windowsazure.com/ProtectionService.svc}
1E2C    1C9C    02/27   15:57:55.514    79  WcfClient.cs(863)       E35FFF34-DF88-4333-A731-0305000562DF        NORMAL  GetStatus Response and Updated variables | Params: {status.LastHearbeatTime = 27/02/2021 15:57:38}{lasttimeClientReceivedNewHeartBeat = 27/02/2021 15:57:38}{lastNewHeatbeat = 27/02/2021 15:57:38}{statusPollTime = 27/02/2021 15:57:55}
1E2C    1C9C    02/27   15:57:55.514    79  WcfClient.cs(815)       E35FFF34-DF88-4333-A731-0305000562DF        WARNING FMBlock: Executing retry policy | Params: {CurrentAttempt = 99}{MaxAttempts = 28800}{BackOffIntervalInSec = 18}{OperationCode = WcfProxyGenericAsyncOperationPollingCall}{Retry policy id = 7}
1E2C    1AD0    02/27   15:57:55.936    79  WcfClient.cs(960)       BD4069C5-779E-4554-82D3-2465A632F306        NORMAL  Executing web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.FileCatalog.Interface.IFileCatalogExternalChannel}{Description = UpdateROArgs}{TargetEndpoint = https://pod01-fc1.ukw.backup.windowsazure.com/FileCatalogExternalService.svc}
1E2C    1AD0    02/27   15:57:55.936    69  CBEngineWcfClientHelper.cs(396)     BD4069C5-779E-4554-82D3-2465A632F306        NORMAL  Setting RequestId header for outgoing request | Params: {RequestId = 88c8b94f-c433-4a08-a117-19f42f49a631}{WorkitemId = ad5657c0-67c8-472c-9734-63b05d306859}{TaskId = bd4069c5-779e-4554-82d3-2465a632f306}{AgentVersion = 2.0.9205.0}
1E2C    099C    02/27   15:57:55.967    71  dscontext.cpp(184)  [000000071BF512D0]  E63383DC-C408-4E3E-A11E-B51218C91F28        ACTIVITY    Last completed state for Ds Id (35184717886556) is 5
1E2C    19B0    02/27   15:57:57.061    79  WcfClient.cs(960)       E63383DC-C408-4E3E-A11E-B51218C91F28        NORMAL  Executing web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.FileCatalog.Interface.IFileCatalogExternalChannel}{Description = UpdateROArgs}{TargetEndpoint = https://pod01-fc1.ukw.backup.windowsazure.com/FileCatalogExternalService.svc}
1E2C    19B0    02/27   15:57:57.061    69  CBEngineWcfClientHelper.cs(396)     E63383DC-C408-4E3E-A11E-B51218C91F28        NORMAL  Setting RequestId header for outgoing request | Params: {RequestId = f5dfafe2-ae69-4339-8f0b-bec26a9262d5}{WorkitemId = d89a37cf-a3f0-4f15-aa3d-20fbc473f3a1}{TaskId = e63383dc-c408-4e3e-a11e-b51218c91f28}{AgentVersion = 2.0.9205.0}
1E2C    1BB8    02/27   15:57:57.123    71  dscontext.cpp(184)  [0000000722B8BB40]  BD4069C5-779E-4554-82D3-2465A632F306        ACTIVITY    Last completed state for Ds Id (35184652646204) is 5
1E2C    1AD0    02/27   15:57:57.311    79  WcfClient.cs(998)       BD4069C5-779E-4554-82D3-2465A632F306        WARNING Unable to make web service call | Params: {Exception:  = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
1E2C    1AD0    02/27   15:57:57.311    79  WcfClient.cs(998)       BD4069C5-779E-4554-82D3-2465A632F306        WARNING  (Fault Detail is equal to ErrorCode = CloudAsyncWorkLimitReached, DetailedErrorCode = 0, DetailedErrorSource = None/None, Message = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
1E2C    1AD0    02/27   15:57:57.311    79  WcfClient.cs(998)       BD4069C5-779E-4554-82D3-2465A632F306        WARNING ).}
1E2C    1AD0    02/27   15:57:57.311    79  WcfClient.cs(946)       BD4069C5-779E-4554-82D3-2465A632F306        WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyCall}{OperationStartTime = 27/02/2021 15:57:55}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
1E2C    1AD0    02/27   15:57:57.311    79  WcfClient.cs(946)       BD4069C5-779E-4554-82D3-2465A632F306        WARNING ]
1E2C    1AD0    02/27   15:57:57.311    79  WcfClient.cs(946)       BD4069C5-779E-4554-82D3-2465A632F306        WARNING {InitialTargetEndpoint = https://pod01-fc1.ukw.backup.windowsazure.com/FileCatalogExternalService.svc}{Name = BasicHttpBinding_IFileCatalogExternal}{WorkitemId = ad5657c0-67c8-472c-9734-63b05d306859}{CurrentTargetEndpoint = https://pod01-fc1.ukw.backup.windowsazure.com/FileCatalogExternalService.svc}{RequestId = 0000000002_88c8b94f-c433-4a08-a117-19f42f49a631_bd4069c5-779e-4554-82d3-2465a632f306}{IdentityType = Container}{Identity = 2629e21830edda7b6e054ae781e7c3441d09d00d}
1E2C    1AD0    02/27   15:57:57.311    79  WcfClient.cs(946)       BD4069C5-779E-4554-82D3-2465A632F306        WARNING }
1E2C    1AD0    02/27   15:57:57.311    79  WcfClient.cs(946)       BD4069C5-779E-4554-82D3-2465A632F306        NORMAL  FMBlock: No retry policy specified | Params: {ErrorCode = CloudAsyncWorkLimitReached}
1E2C    1AD0    02/27   15:57:57.311    79  WcfClient.cs(946)       BD4069C5-779E-4554-82D3-2465A632F306        NORMAL  FMBlock: Mapping source error to target error | Params: {OperationCode = WcfProxyCall}{SourceError = CloudAsyncWorkLimitReached}{TargetError = CloudAsyncWorkLimitReached}
1E2C    1AD0    02/27   15:57:57.311    79  WcfClient.cs(694)       BD4069C5-779E-4554-82D3-2465A632F306        WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyAsyncCall}{OperationStartTime = 27/02/2021 15:53:10}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
1E2C    1AD0    02/27   15:57:57.311    79  WcfClient.cs(694)       BD4069C5-779E-4554-82D3-2465A632F306        WARNING ]
1E2C    1AD0    02/27   15:57:57.311    79  WcfClient.cs(694)       BD4069C5-779E-4554-82D3-2465A632F306        WARNING {InitialTargetEndpoint = https://pod01-fc1.ukw.backup.windowsazure.com/FileCatalogExternalService.svc}{Name = BasicHttpBinding_IFileCatalogExternal}{WorkitemId = ad5657c0-67c8-472c-9734-63b05d306859}{CurrentTargetEndpoint = https://pod01-fc1.ukw.backup.windowsazure.com/FileCatalogExternalService.svc}{RequestId = 0000000002_88c8b94f-c433-4a08-a117-19f42f49a631_bd4069c5-779e-4554-82d3-2465a632f306}{IdentityType = Container}{Identity = 2629e21830edda7b6e054ae781e7c3441d09d00d}
1E2C    1AD0    02/27   15:57:57.311    79  WcfClient.cs(694)       BD4069C5-779E-4554-82D3-2465A632F306        WARNING }
1E2C    1AD0    02/27   15:57:57.311    79  WcfClient.cs(694)       BD4069C5-779E-4554-82D3-2465A632F306        WARNING FMBlock: Current operationTime timeout reached or will reach after backoff time. | Params: {OperationCode = WcfProxyAsyncCall}{OperationTimeoutTime = 27/02/2021 15:58:10}{BackOffTimeInSec = 18}{CurrentAttempt = 16}{MaxAttempts = 125}
1E2C    1AD0    02/27   15:57:57.311    79  WcfClient.cs(694)       BD4069C5-779E-4554-82D3-2465A632F306        NORMAL  FMBlock: Mapping source error to target error | Params: {OperationCode = WcfProxyAsyncCall}{SourceError = CloudAsyncWorkLimitReached}{TargetError = CloudTimeoutError}
1E2C    1AD0    02/27   15:57:57.311    69  CalExternalServiceProxy.cs(2732)        BD4069C5-779E-4554-82D3-2465A632F306        WARNING FMBlock: FM exception caught | Params: {OperationCode = UpdateROArgs}{OperationStartTime = 27/02/2021 15:53:1

Share via

Having problem backing up a file disk using DPM to the cloud.

1 answer

Your answer