Is anyone else having problems with high CPU consumption using (azure workload backup)?

Question

Is anyone else having problems with high CPU consumption using (azure workload backup)?

Fernando R Silva 80

Mar 1, 2024, 1:05 PM

Since 02/26/2024 I have been facing a problem with performance on all my Azure VMs. I was able to identify that the problem starts with the service (azure workload backup) which started to consume all the cpu. After uninstalling it, processing returns to normal. However, this service is responsible for the automatic backup of the SQL Server databases via Azure, that is, if I leave the VM running, it freezes, if I uninstall it I will be left without a backup of the databases. Has anyone experienced the problem? How did you solve it?

I can't wait, thank you.

Dbla 10 Reputation points

Mar 3, 2024, 11:43 AM

I had the same thing popping up since yesterday on all of my azure VMs that have an MSSQL server installed (and backup through azure backup for those DBs)

not just have they high CPU load, but it seems they are also hammeraging the domain controllers with endless requests, which leads also to high CPU load on their end.

i can't tell you a solution yet as i'm also just starting to take a look at it, but you are not alone.

for me it looks like the azure backup vm-agent got screwed up with an update
Marco Böhme 5 Reputation points

Mar 4, 2024, 9:01 AM

We've got the same problem.

The "backup" started for some servers which had databases in the past but not anymore. It triggered the job because of them still being configured in the backup vault.

But we also have got some databases which are still today (after 3 days) doing a backup.

We can also see that the Domain Controllers are hammered with requests and have got a high load because of this.

The problem for us is, that we don't know why these backup jobs get created and are running (there is no backup job for this time in the day visible). Furthermore we don't know where these backups are going, because when there is no task to create these backups there is no task to use/show/delete this data.

It is costing us huge amounts of dollars in network traffic to this day and there is no clear communication from microsoft to this point.

Regards
Tim-5081 120 Reputation points

Mar 4, 2024, 3:39 PM

We're facing the exact same issue within our tenant. Besides the huge amount of costs for networktraffic we also see highly degraded performance of our applications running on the VMs. Sometimes we need to restart them to continue working. The only workaround right now is to uninstall the VM extension. As the Workload Backup (SQL) is not working then, this is not an acceptable solution, so that we have to live with the massive performance problems and costs. I opened an MS-Ticket for this issue, but right now we're waiting for the answer.
David-Alexandre CAUBET 25 Reputation points

Mar 5, 2024, 8:37 PM

Hello,

I am also encountering this problem since the same day.

Upon analysis, we have identified the same thing, an increase in CPU load to saturation at the time of SQL backup in Azure VMs.

The iaaSWLPluginSvc process along with LSASS.exe are taking up all the CPU resources of the VMs. The LSASS.exe process does not decrease and I also have a domain controller VM on which the CPU remains saturated with the LSASS.exe process maxed out!

Does anyone have a viable solution? Without stopping the backups!

We have opened a Microsoft ticket but they have not provided an answer so far, at least!
Tim-5081 120 Reputation points

Mar 6, 2024, 6:57 AM

@David-Alexandre CAUBET Have I got it right, that the issue for you is only present while the backup job is running? Right now, we know no other solution than stopping the backup. We've also opened an MS-Ticket but unfortunately we also have no answer so far.
David-Alexandre CAUBET 25 Reputation points

Mar 6, 2024, 11:16 AM

Hello,

Thank you fdor the reply :)

We have conducted a test by uninstalling the Azure Backup extension on test VMs, and the observation is immediate: the CPU load returns to normal right after uninstallation.

Therefore, there is indeed an issue on the Azure platform with SQL backup jobs in the VMs.

However, as of now, there is no communication from Microsoft on this matter, which is critical for data security!!!
Tim-5081 120 Reputation points

Mar 6, 2024, 12:11 PM

@David-Alexandre CAUBET You're welcome.

We're seeing the same issue as described by you. We have escalated the ticket at MS but still waiting for an answer. I agree that this is a very critical issue. We keep you updated here.
Fernando R Silva 80 Reputation points

Mar 6, 2024, 1:09 PM

I deployed Azure Workload Backup for testing today and the problem persists

We still have no response from microsft.

We have more than seventy bases on the server, it is unfeasible to do this task manually.

If anything new comes up, I'll let you know here.
Tim-5081 120 Reputation points

Mar 7, 2024, 6:43 AM

Quick update on this case: Microsoft was able to find an issue within the provided log files. Right now, it seems to be an authentication issue between the extension and the SQL server. Our ticket was escalated to the product group, and we're now awaiting their answer.
Fernando R Silva 80 Reputation points

Mar 7, 2024, 10:09 AM

Thanks @Tim-5081, your ticket appears to be the most advanced. When you receive a response, please let us know. Thanks
Vincent Alaminos 5 Reputation points

Mar 7, 2024, 3:16 PM

Hi, I'm facing the same issue as you only with SQL reporting server. Did you have any news or solutions from MS ?
Tim-5081 120 Reputation points

Mar 7, 2024, 3:19 PM

I will keep you updated. But unfortunately, nothing there is nothing new right now.
Tim-5081 120 Reputation points

Mar 7, 2024, 4:48 PM

@Vincent Alaminos Yes, see the answer provided by me.
Ralph J. Möller | ant.at 5 Reputation points

Mar 11, 2024, 8:24 AM

Hi, same issue here. Will now try Tim's answer and get back with result.

Accepted answer

1 additional answer

Your answer

Dbla 10 Reputation points

Mar 3, 2024, 11:43 AM

I had the same thing popping up since yesterday on all of my azure VMs that have an MSSQL server installed (and backup through azure backup for those DBs)

not just have they high CPU load, but it seems they are also hammeraging the domain controllers with endless requests, which leads also to high CPU load on their end.

i can't tell you a solution yet as i'm also just starting to take a look at it, but you are not alone.

for me it looks like the azure backup vm-agent got screwed up with an update
Marco Böhme 5 Reputation points

Mar 4, 2024, 9:01 AM

We've got the same problem.

The "backup" started for some servers which had databases in the past but not anymore. It triggered the job because of them still being configured in the backup vault.

But we also have got some databases which are still today (after 3 days) doing a backup.

We can also see that the Domain Controllers are hammered with requests and have got a high load because of this.

The problem for us is, that we don't know why these backup jobs get created and are running (there is no backup job for this time in the day visible). Furthermore we don't know where these backups are going, because when there is no task to create these backups there is no task to use/show/delete this data.

It is costing us huge amounts of dollars in network traffic to this day and there is no clear communication from microsoft to this point.

Regards
Tim-5081 120 Reputation points

Mar 4, 2024, 3:39 PM

We're facing the exact same issue within our tenant. Besides the huge amount of costs for networktraffic we also see highly degraded performance of our applications running on the VMs. Sometimes we need to restart them to continue working. The only workaround right now is to uninstall the VM extension. As the Workload Backup (SQL) is not working then, this is not an acceptable solution, so that we have to live with the massive performance problems and costs. I opened an MS-Ticket for this issue, but right now we're waiting for the answer.
David-Alexandre CAUBET 25 Reputation points

Mar 5, 2024, 8:37 PM

Hello,

I am also encountering this problem since the same day.

Upon analysis, we have identified the same thing, an increase in CPU load to saturation at the time of SQL backup in Azure VMs.

The iaaSWLPluginSvc process along with LSASS.exe are taking up all the CPU resources of the VMs. The LSASS.exe process does not decrease and I also have a domain controller VM on which the CPU remains saturated with the LSASS.exe process maxed out!

Does anyone have a viable solution? Without stopping the backups!

We have opened a Microsoft ticket but they have not provided an answer so far, at least!
Tim-5081 120 Reputation points

Mar 6, 2024, 6:57 AM

@David-Alexandre CAUBET Have I got it right, that the issue for you is only present while the backup job is running? Right now, we know no other solution than stopping the backup. We've also opened an MS-Ticket but unfortunately we also have no answer so far.
David-Alexandre CAUBET 25 Reputation points

Mar 6, 2024, 11:16 AM

Hello,

Thank you fdor the reply :)

We have conducted a test by uninstalling the Azure Backup extension on test VMs, and the observation is immediate: the CPU load returns to normal right after uninstallation.

Therefore, there is indeed an issue on the Azure platform with SQL backup jobs in the VMs.

However, as of now, there is no communication from Microsoft on this matter, which is critical for data security!!!
Tim-5081 120 Reputation points

Mar 6, 2024, 12:11 PM

@David-Alexandre CAUBET You're welcome.

We're seeing the same issue as described by you. We have escalated the ticket at MS but still waiting for an answer. I agree that this is a very critical issue. We keep you updated here.
Fernando R Silva 80 Reputation points

Mar 6, 2024, 1:09 PM

I deployed Azure Workload Backup for testing today and the problem persists

We still have no response from microsft.

We have more than seventy bases on the server, it is unfeasible to do this task manually.

If anything new comes up, I'll let you know here.
Tim-5081 120 Reputation points

Mar 7, 2024, 6:43 AM

Quick update on this case: Microsoft was able to find an issue within the provided log files. Right now, it seems to be an authentication issue between the extension and the SQL server. Our ticket was escalated to the product group, and we're now awaiting their answer.
Fernando R Silva 80 Reputation points

Mar 7, 2024, 10:09 AM

Thanks @Tim-5081, your ticket appears to be the most advanced. When you receive a response, please let us know. Thanks
Vincent Alaminos 5 Reputation points

Mar 7, 2024, 3:16 PM

Hi, I'm facing the same issue as you only with SQL reporting server. Did you have any news or solutions from MS ?
Tim-5081 120 Reputation points

Mar 7, 2024, 3:19 PM

I will keep you updated. But unfortunately, nothing there is nothing new right now.
Tim-5081 120 Reputation points

Mar 7, 2024, 4:48 PM

@Vincent Alaminos Yes, see the answer provided by me.
Ralph J. Möller | ant.at 5 Reputation points

Mar 11, 2024, 8:24 AM

Hi, same issue here. Will now try Tim's answer and get back with result.

Answer 1

Tim-5081 120

Mar 7, 2024, 4:03 PM

MS has provided us the following Workaround (seems to work for us by now):

go to C:\Program Files\Azure Workload Backup\bin\plugins\SQL and open PluginManifest.json
Change the value of AllowCommunicationThroughNamedPipes from true to false
Open TaskManager and Stop AzureWLBackupPluginSvc and AzureWLBackupInquirySvc. Restart AzureWLBackupCoordinatorSvc.

Just try and give Feedback if this is helping you. We will let you know if we have further information on the root cause.

Alexandre V. Carneiro 5 Reputation points

Mar 7, 2024, 5:29 PM

We received the same workaround from Microsoft and so far it is working.

Thank you for sharing!
Fernando R Silva 80 Reputation points

Mar 7, 2024, 5:30 PM

I applied your tip and it seems to be working normally now.

I will apply the configuration to a server with many databases and monitor the backups and integrity.

I will post the result in the next 48 hours

Thank you very much for your efforts in resolving the problem. Congratulations!!!
Fernando R Silva 80 Reputation points

Mar 7, 2024, 5:33 PM

Thanks @Tim-5081
Vincent Alaminos 5 Reputation points

Mar 8, 2024, 8:38 AM

Thanks for sharing ! It is working normallly for now.
Stéphane LEMOULE 5 Reputation points

Mar 8, 2024, 10:36 AM

It seems to have fix the issue on our side. Thanks for sharing.
David-Alexandre CAUBET 25 Reputation points

Mar 8, 2024, 3:34 PM

Thanks for sharing.

Fix applied on several test VMs and SQL backup successfully restarted without CPU increase.

For now the solution seems viable and stable!
Tim-5081 120 Reputation points

Mar 11, 2024, 11:55 AM

We got new information from Microsoft on that. An updated extension Version (1.1.0.94) should be deployed to all regions by 12th March EOD. Then the behavior should return to normal without the workaround.
Fernando R Silva 80 Reputation points

Mar 11, 2024, 12:26 PM

Good morning everyone, today it's been three days since we reactivated the backup following @Tim-5081's tip and everything is working fine.

We are considering the case as resolved.

The fear is, when will Microsoft send another one like this?
Ralph J. Möller | ant.at 5 Reputation points

Apr 2, 2024, 2:30 PM

I can report that the solution worked for me. Had severe issues, after the patch everything back to normal.
Tim-5081 120 Reputation points

Apr 2, 2024, 3:12 PM

Thanks for your feedback, Ralph. We also didn't see any issues after the patch.

Answer 2

Sedat SALMAN 14,160 MVP

Mar 2, 2024, 7:54 AM

Use Azure's PerfInsights to check why your CPU usage is so high. It helps you figure out if Azure's backup service is the problem

https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/troubleshoot-high-cpu-issues-azure-windows-vm

After running the tool, look at the report. It tells you what's using the most CPU. If the backup service is the issue, it should show up here

Fernando R Silva 80 Reputation points

Mar 2, 2024, 11:17 AM

Hello Sedat SALMAN, thanks for the answer.

I have already identified the cause and it is actually in the Azure backup service.

Below is an image before installing the backup service

After install

Below is also an image of the task manager

And this service worked perfectly until 02/26/2024 and started to present this problem.

The same case occurs on Windows server 2016 as on Windows server 2019.

Thanks.
Tim-5081 120 Reputation points

Mar 4, 2024, 3:32 PM

We're facing the same issue as written above. We were able to shurly identify, that the Azure Workload Backup service is causing the issue. As we delete the extension or stopping the services, the behavior described above immediately stops.
Fernando R Silva 80 Reputation points

Mar 4, 2024, 5:34 PM

In our case, excluding just the extension, when we stop the service it is activated automatically because of the tasks scheduled by the service itself.

We continue without a look from Microsoft.

Another point that I noticed after this update is, the processing of IIS websites, when it exceeds 50 connections the CPU also goes to 100%.
Tim-5081 120 Reputation points

Mar 5, 2024, 8:20 AM

@Fernando R Silva Yeah, right - the services starting again if they're only disabled. For that, I've tried to uninstall the extension. This solves the problem "long-term". We don't have IIS on our servers, so that we can't see this behavior.
Tim-5081 120 Reputation points

Mar 5, 2024, 8:21 AM

Deleted double entry

AI Skills Fest

Share via

Is anyone else having problems with high CPU consumption using (azure workload backup)?

1 additional answer

Your answer