How to reliably attach data disks in VMSS CustomScript extension

I'm trying to make a VMSS with persistent data disks and developed a PowerShell script to be executed in the CustomScript extension.

However, it occasionally fails to attach a disk to a VM instance when scaling up occurs with 4 or more instances at a time. I've investigated the problem and tried to mitigate the possible reasons but no luck so far.

Here's the explanation of the problem and its background project. It includes the ARM template with step by setup guide so that you can easily reproduce the problem.

https://dev.azure.com/banademo/CNCoCommon/_wiki/wikis/CNCoCommon.wiki/7

Could someone please tell me any insights about it?

1 answer

shiva patpi 13,131 Reputation points Microsoft Employee

2021-11-24T07:38:48.677+00:00
Hello @八重樫剛史 Takeshi Yaegashi ,
Thanks for the detailed description of the problem statement.
Went through the setup.ps1 script, basically the problem is with $LUN variable which is passed as a parameter which will always have the value of 0. That variable value needs to be incremented by 1 every time when you try to attach a new data disk. Since it is trying to use the same number hence it is complaining about the conflict.

Example Repro:

When we run the command:
Add-AzvmssVMDataDisk -VirtualMachineScaleSetVM $vm -ManagedDiskId $disk.Id -Caching "ReadOnly" -CreateOption Attach -Lun 0 | Update-AzvmssVM

It will add and attach the data disk with LUN 0

Now If i run the same command for the second disk

Add-AzvmssVMDataDisk -VirtualMachineScaleSetVM $vm -ManagedDiskId $disk.Id -Caching "ReadOnly" -CreateOption Attach -Lun 0 | Update-AzvmssVM
Update-AzvmssVM : A disk named 'testdisk1' already uses the same LUN: 0.
ErrorCode: InvalidParameter
ErrorMessage: A disk named 'testdisk1' already uses the same LUN: 0.
ErrorTarget: dataDisk.lun
StatusCode: 400
ReasonPhrase: Bad Request
OperationID : 831f1bcf-7fcb-4922-9d6f-00eef6d15fb5
At line:1 char:130

... -Caching "ReadOnly" -CreateOption Attach -Lun 0 | Update-AzvmssVM

~~~~~~~~~~~~~~~

CategoryInfo : CloseError: (:) [Update-AzVmssVM], ComputeCloudException

FullyQualifiedErrorId : Microsoft.Azure.Commands.Compute.Automation.UpdateAzureRmVmssVM

///////////////////////////////////

Now If i change the last command to use LUN 1

Add-AzvmssVMDataDisk -VirtualMachineScaleSetVM $vm -ManagedDiskId $disk.Id -Caching "ReadOnly" -CreateOption Attach -Lun 1 | Update-AzvmssVM

Mitigation:

So the fix for your script should be to increment the value of LUN every time when a new data disk gets added before adding the next disk.

All the very best !!

If the above details help out in resolving your issue , please make sure to "Upvote & Accept the answer"

Regards,
Shiva.
Please sign in to rate this answer.
八重樫剛史 Takeshi Yaegashi 1 Reputation point

2021-11-24T10:11:21.48+00:00

Hi @shiva patpi , thanks for the quick response.

Yes, the script fails again and again with the LUN 0 conflict errors, but it's just the consequence of the real problem I want to solve.

I don't want to change the LUN for retry attempts because it can cause more complicated situations like multiple disks attached and inconsistent disk numbers on Windows OS.

The best way would be to detach the fault disk to clear LUN 0 before moving on to the next attempt. But it seems the CustomScript session (setup.ps1 itself) must be terminated to detach disks. So all it can do would be just wait for LUN 0 to be vacant.

shiva patpi 13,131 Reputation points Microsoft Employee

2021-11-26T21:23:10.88+00:00

When I click on button "Deploy To Azure", resource group, VNET, NSG got created.
VMSS also got created but the Instances are not getting created. So I am not able to repro locally.

八重樫剛史 Takeshi Yaegashi 1 Reputation point

2021-11-27T06:08:09.497+00:00

Thanks for looking into this.

Yes, the initial number of the VMSS instances are set to 0 in the template.
You need to manually increase the instance count in Azure Portal to reproduce the problem.

Please refer How to reproduce the problem guide.
I strongly recommend you to arrange the deployment parameters to reduce the cost and avoid hitting quota.
Set the instance count to 4 and see if all of CustomScript extension (csextension) deployments terminate successfully.
You will see the inconsistency of attached/unattached status between VM and disk resources on failed instances.

shiva patpi 13,131 Reputation points Microsoft Employee

2021-11-27T17:38:34.37+00:00

I just tried deploying using initial 4 instances. After deploying data disks were not attached to any of instances and Latest Model was showing as "NO".
Then I tried updating instance3 , after which data disk got attached successfully.
Then I tried adding 2 additional instances which got created successfully along with corresponding data disks.

Example of data disk attached successfully:

No logs in the analytics workspace

No errors in the custom script extension:

Now the most probably question/problem to be solved is: - Why initial data disks were not attached to those 4 instances after deploying?

八重樫剛史 Takeshi Yaegashi 1 Reputation point

2021-11-27T18:07:37.623+00:00

The first 4 instances would have failed in setup.ps1 for some reason. You should check the Windows event logs on the instances.
Did you connect the VMSS to a Log Anaylytics workspace and enable Windows event logs for "Application" in the agent configuration?
If it's set up correctly, a simple query "Event" shows you many event logs other than "DataPersist" source.

You should add 4 or more instances at a time in order to reproduce the problem.
Adding instances less than 4 will successfully create and mount data disks.

shiva patpi 13,131 Reputation points Microsoft Employee

2021-11-28T00:30:00.337+00:00

I don't see any issues.
Initially deployed with 0 instances. Scaled up to 6 instances which are perfectly spined-up along with corresponding data disks. Then scaled-up again with 6 more instances which again went through without any issues.

八重樫剛史 Takeshi Yaegashi 1 Reputation point

2021-11-28T00:59:01.973+00:00

It's worked in your environment? That sounds great!
So it might be a region-specific problem or something like that. Could you share in which region you deployed it?
I'll test further in regions other than Japan East, and if the problem persists, consult the paid Azure support this week.

Thanks for consuming your time to join the test and investigation. I appreciate it.

shiva patpi 13,131 Reputation points Microsoft Employee

2021-11-28T01:00:21.997+00:00

I tried in EastUS region

八重樫剛史 Takeshi Yaegashi 1 Reputation point

2021-11-28T01:58:06.17+00:00

I've tried in East US and West US2 regions and still got the same result.
In both regions, only the first 3 instances succeeded to create and mount data disks.
Other instances failed. They look Running and OK in the instance listing, but each of them has an error in the deployment of the CustomScript extension.

I've tried with the following parameters:

{ "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#", "contentVersion": "1.0.0.0", "parameters": { "vmssName": { "value": "vmss4" }, "vmSize": { "value": "Standard_D2as_v4" }, "vmImageWindowsSKU": { "value": "2019-Datacenter-with-Containers-smalldisk" }, "vmDiskPlacement": { "value": "CacheDisk" }, "vmDiskSizeGB": { "value": 50 }, "vmSubnetId": { "value": "" }, "vmTimeZone": { "value": "Tokyo Standard Time" }, "vmSetupArgs": { "value": "-Expand -Sku Premium_LRS -SizeGB 64" }, "adminUsername": { "value": "azureuser" }, "adminPassword": { "value": null } } }
Sign in to comment