Hi @JF ,
Welcome to the Microsoft Q&A platform. I don't think the issue is caused from removing the BGInfo extension. BGInfo is an extension that writes useful server information to the desktop as a wallpaper. If you create VMs from Portal you already get this extension. Once Bginfo is enabled, the VM desktop has a wallpaper like this:
Reference : start-azvm
I would suggest trying the powershell way of starting the VM using below command
Start-AzVM -ResourceGroupName "ResourceGroupxxx" -Name "Vmxxx"
If that doesn' help, you can take a snapshot of the os disk and create a disk from the snapshot and use it to create a VM from portal. Here is the document with the details, reference : create-vm-specialized-portal
Please "Accept as Answer" and Upvote if any of above helped so that, it can help others in community looking for remediation for the similar issues.