How to create resources at scale using the Azure CLI
Article
As an Azure resource manager, you frequently have to create multiple Azure resources when configuring new environments. You might also have an Azure resource approval process that works best when Azure resources are created automatically from a script.
In this article you will learn the following:
Create multiple Azure resources from parameters received from a delimited CSV file.
Use IF..THEN statements to create dependent Azure resources.
Download and save to a local directory the following CSV file. Replace myExistingResourceGroupName in line three with an actual resource group name.
resourceNo,location,createRG,exstingRgName,createVnet,vnetAddressPrefix,subnetAddressPrefixes,vmImage,publicIpSku,Adminuser
1,eastus,TRUE,,TRUE,10.0.0.0/16,10.0.0.0/24,Ubuntu2204,standard,
2,eastus2,TRUE,,FALSE,,,Debian11,standard,alex-smith
3,southcentralus,FALSE,myExistingResourceGroupName,FALSE,,,Ubuntu2204,standard,jan-smith
[empty line for Bash]
Note
To be a proper Unix text file and be read by Bash, the CSV file needs a newline character at the end of the last data line. This results in a blank line at the end of the file. Your blank line does not need to say [empty line] as this text is only provided to show you that an empty line exists. PowerShell environments do not have this newline character requirement.
Upload your modified CSV file to your Azure Cloud Shell blog storage account. The easiest way to do this is to use the Manage files drop down on the Azure Cloud Shell main menu. For more information on Cloud Shell storage, see Persist files in Azure Cloud Shell.
Script overview
This article breaks a single large script into four sections so each step can be explained.
Variable setup
Data validation
Loop validation
Azure resource creation
There are also two scripts provided: one for Bash and the second for PowerShell. Both scripts use the same Azure CLI commands. It is the environment, or terminal profile, that is different. For example, Bash uses do...done and if...then...fi. In a PowerShell environment, you use the equivalent foreach and if (something is true)...{do this}. In Azure Cloud Shell you can switch between environments by using the Switch to PowerShell or Switch to Bash button in the Azure Cloud Shell main menu.
Get started by creating the variables needed for the script. The following three variables need actual values for your environment:
subscriptionID: This is your Azure subscription ID.
csvFileLocation: This is the location and file name of your CSV input file.
logFileLocation: This is the location and file name the script will use to create a log file. You do not need to create or upload this file.
Variables with a msdocs- prefix can be replaced with the prefix of your choice. All empty ("") variables use values from the CSV input file. These empty variables are placeholders needed by the script.
# Variable block
# Replace these three variable values with actual values
subscriptionID=00000000-0000-0000-0000-00000000
csvFileLocation="myFilePath\myFileName.csv"
logFileLocation="myFilePath\myLogName.txt"
# Variable values that contain a prefix can be replaced with the prefix of your choice.
# These prefixes have a random ID appended to them in the script.
# Variable values without a prefix will be overwritten by the contents of your CSV file.
location=""
createRG=""
newRgName="msdocs-rg-"
existingRgName=""
createVnet=""
vnetName="msdocs-vnet-"
subnetName="msdocs-subnet-"
vnetAddressPrefix=""
subnetAddressPrefixes=""
vmName="msdocs-vm-"
vmImage=""
publicIpSku=""
adminUser=""
adminPassword="msdocs-PW-@"
# Set your Azure subscription
az account set --subscription $subscriptionID
# Variable block
# Replace these three variable values with actual values
$subscriptionID = "00000000-0000-0000-0000-00000000"
$csvFileLocation = "myFilePath\myFileName.csv"
$logFileLocation = "myFilePath\myLogName.txt"
# Variable values that contain a prefix can be replaced with the prefix of your choice.
# These prefixes have a random ID appended to them in the script.
# Variable values without a prefix will be overwritten by the contents of your CSV file.
$location=""
$createRG=""
$newRgName="msdocs-rg-"
$existingRgName=""
$createVnet=""
$vnetName="msdocs-vnet-"
$subnetName="msdocs-subnet-"
$vnetAddressPrefix=""
$subnetAddressPrefixes=""
$vmName="msdocs-vm-"
$vmImage=""
$publicIpSku=""
$adminUser=""
$adminPassword="msdocs-PW-@"
# Set your azure subscription
az account set --subscription $subscriptionID
# Import your CSV data
$data = Import-Csv $csvFileLocation -delimiter ","
Validate CSV file values
Before you start to test the create script, make sure your CSV file is formatted correctly and variables will be assigned correct values. This script uses an IF..THEN statement so you can look at one scenario/CSV line at a time.
# Verify CSV columns are being read correctly
# Take a look at the CSV contents
cat $csvFileLocation
# Validate select CSV row values
while IFS=, read -r resourceNo location createRG existingRgName createVnet vnetAddressPrefix subnetAddressPrefixes vmImage publicIpSku adminUser
do
# Generate a random ID
let "randomIdentifier=$RANDOM*$RANDOM"
# Return the values for the first data row
# Change the $resourceNo to check different scenarios in your CSV
if [ "$resourceNo" = "1" ]; then
echo "resourceNo = $resourceNo"
echo "location = $location"
echo "randomIdentifier = $randomIdentifier"
echo ""
echo "RESOURCE GROUP INFORMATION:"
echo "createRG = $createRG"
if [ "$createRG" = "TRUE" ]; then
echo "newRGName = $newRgName$randomIdentifier"
else
echo "exsitingRgName = $existingRgName"
fi
echo ""
echo "VNET INFORMATION:"
echo "createVnet = $createVnet"
if [ "$createVnet" = "TRUE" ]; then
echo "vnetName = $vnetName$randomIdentifier"
echo "subnetName = $subnetName$randomIdentifier"
echo "vnetAddressPrefix = $vnetAddressPrefix"
echo "subnetAddressPrefixes = $subnetAddressPrefixes"
fi
echo ""
echo "VM INFORMATION:"
echo "vmName = $vmName$randomIdentifier"
echo "vmImage = $vmImage"
echo "vmSku = $publicIpSku"
if [ `expr length "$adminUser"` == "1" ]; then
echo "SSH keys will be generated."
else
echo "vmAdminUser = $adminUser"
echo "vmAdminPassword = $adminPassword$randomIdentifier"
fi
fi
# skip the header line
done < <(tail -n +2 $csvFileLocation)
# Verify CSV columns are being read correctly
# Take a look at the CSV contents
# The returned table is restricted by the width of your terminal window
$data | Format-Table
# Validate select CSV row values
foreach ($row in $data) {
$resourceNo = $row.resourceNo
$location = $row.location
$createRG = $row.createRG
$existingRgName = $row.existingRgName
$createVnet = $row.createVnet
$vnetAddressPrefix = $row.vnetAddressPrefix
$subnetAddressPrefixes = $row.subnetAddressPrefixes
$vmImage = $row.vmImage
$publicIpSku = $row.publicIpSku
$adminUser = $row.adminUser
# Generate a random ID
$randomIdentifier = (New-Guid).ToString().Substring(0,8)
# Return the values for the first data row
# Change the $resourceNo to check different scenarios in your CSV
if ($resourceNo -eq "1") {
Write-Host "resourceNo = $resourceNo"
Write-Host "location = $location"
Write-Host "randomIdentifier = $randomIdentifier"
Write-Host ""
Write-Host "RESOURCE GROUP INFORMATION:"
Write-Host "createRG = $createRG"
if ($createRG -eq "TRUE") {
Write-Host "newRGName = $newRgName$randomIdentifier"
}
else {
Write-Host "exsitingRgName = $existingRgName"
}
Write-Host ""
Write-Host "VNET INFORMATION:"
Write-Host "createVnet = $createVnet"
if ($createVnet -eq "TRUE") {
Write-Host "vnetName = $vnetName$randomIdentifier"
Write-Host "subnetName = $subnetName$randomIdentifier"
Write-Host "vnetAddressPrefix = $vnetAddressPrefix"
Write-Host "subnetAddressPrefixes = $subnetAddressPrefixes"
}
Write-Host ""
Write-Host "VM INFORMATION:"
Write-Host "vmName = $vmName$randomIdentifier"
Write-Host "vmImage = $vmImage"
Write-Host "vmSku= $publicIpSku"
if ($adminUser -ne "") {
Write-Host "vmAdminUser = $adminUser"
Write-Host "vmAdminPassword = $adminPassword$randomIdentifier"
}
else {
Write-Host "SSH keys will be generated."
}
}
}
Using the CSV provided in this article, the validation output is as follows: (The 00000001 random ID will be different for each test.)
resourceNo = 1
location = eastus
RESOURCE GROUP INFORMATION:
createRG = TRUE
newRGName = msdocs-rg-00000001
VNET INFORMATION:
createVnet = TRUE
vnetName = msdocs-vnet-00000001
subnetName = msdocs-subnet-00000001
vnetAddressPrefix = 10.0.0.0/16
subnetAddressPrefix = 10.0.0.0/24
VM INFORMATION:
vmName = msdocs-vm-00000001
vmImage = Ubuntu2204
vmSku = standard
SSH keys will be created
Validate script logic
If you are confident in your scripting abilities, you can skip this step. However, because this script is designed to create Azure resources at scale, looping through the script with echo or write-host statements can save you time and unexpected billable Azure resources.
There are several ways to iterate through a CSV file using Bash. This example uses IFS with a while loop.
# Validate script logic
# Create the log file
echo "SCRIPT LOGIC VALIDATION.">$logFileLocation
# Loop through each row in the CSV file
while IFS=, read -r resourceNo location createRG existingRgName createVnet vnetAddressPrefix subnetAddressPrefixes vmImage publicIpSku adminUser
do
# Generate a random ID
let "randomIdentifier=$RANDOM*$RANDOM"
# Log resource number and random ID
echo "resourceNo = $resourceNo">>$logFileLocation
echo "randomIdentifier = $randomIdentifier">>$logFileLocation
# Check if a new resource group should be created
if [ "$createRG" == "TRUE" ]; then
echo "Will create RG $newRgName$randomIdentifier.">>$logFileLocation
existingRgName=$newRgName$randomIdentifier
fi
# Check if a new virtual network should be created, then create the VM
if [ "$createVnet" == "TRUE" ]; then
echo "Will create VNet $vnetName$randomIdentifier in RG $existingRgName.">>$logFileLocation
echo "Will create VM $vmName$randomIdentifier in Vnet $vnetName$randomIdentifier in RG $existingRgName.">>$logFileLocation
else
echo "Will create VM $vmName$randomIdentifier in RG $existingRgName.">>$logFileLocation
fi
# Skip the header line.
done < <(tail -n +2 $csvFileLocation)
# Clear the console and display the log file
Clear
cat $logFileLocation
# Validate script logic
# Create the log file
"SCRIPT LOGIC VALIDATION." | Out-File -FilePath $logFileLocation
# Loop through each row in the CSV file
foreach ($row in $data) {
$resourceNo = $row.resourceNo
$location = $row.location
$createRG = $row.createRG
$existingRgName = $row.existingRgName
$createVnet = $row.createVnet
$vnetAddressPrefix = $row.vnetAddressPrefix
$subnetAddressPrefixes = $row.subnetAddressPrefixes
$vmImage = $row.vmImage
$publicIpSku = $row.publicIpSku
$adminUser = $row.adminUser
# Generate a random ID
$randomIdentifier = (New-Guid).ToString().Substring(0,8)
# Log resource number and random ID
"" | Out-File -FilePath $logFileLocation -Append
"resourceNo = $resourceNo" | Out-File -FilePath $logFileLocation -Append
"randomIdentifier = $randomIdentifier" | Out-File -FilePath $logFileLocation -Append
# Check if a new resource group should be created
if ($createRG -eq "TRUE") {
"Will create RG $newRgName$randomIdentifier." | Out-File -FilePath $logFileLocation -Append
$existingRgName = "$newRgName$randomIdentifier"
}
# Check if a new virtual network should be created, then create the VM
if ($createVnet -eq "TRUE") {
"Will create VNet $vnetName$randomIdentifier in RG $existingRgName." | Out-File -FilePath $logFileLocation -Append
"Will create VM $vmName$randomIdentifier in VNet $vnetName$randomIdentifier in RG $existingRgName." | Out-File -FilePath $logFileLocation -Append
} else {
"Will create VM $vmName$randomIdentifier in RG $existingRgName." | Out-File -FilePath $logFileLocation -Append
}
}
# Display the log file
Get-Content -Path $logFileLocation
Using the CSV provided in this article, the validation output is as follows: (The 00000001, 2, 3 random IDs will be different for each test, but each resource under each resourceNo should share the same random ID.)
resourceNo = 1
createRG = TRUE
createVnet = TRUE
Will create RG msdocs-rg-00000001
Will create VNet msdocs-vnet-00000001 in RG msdocs-rg-00000001
Will create VM msdocs-vm-00000001 within Vnet msdocs-vnet-00000001 in RG msdocs-rg-00000001
resourceNo = 2
createRG = TRUE
createVnet = FALSE
Will create RG msdocs-rg-00000002
Will create VM msdocs-vm-00000002 without Vnet in RG msdocs-rg-00000002
resourceNo = 3
createRG = FALSE
createVnet = FALSE
Will create VM msdocs-vm-00000003 without Vnet in RG <myExistingResourceGroup>
Create Azure resources
You have now created your variable block, validated your CSV values, and completed a test run with echo or write-host. Execute the fourth and final portion of the script to create Azure resources as defined in your CSV input file.
# Create Azure resources
# Create the log file
echo "CREATE AZURE RESOURCES.">$logFileLocation
# Loop through each CSV row
while IFS=, read -r resourceNo location createRG existingRgName createVnet vnetAddressPrefix subnetAddressPrefixes vmImage publicIpSku adminUser
do
# Generate a random ID
let "randomIdentifier=$RANDOM*$RANDOM"
# Log resource number, random ID and display start time
echo "resourceNo = $resourceNo">>$logFileLocation
echo "randomIdentifier = $randomIdentifier">>$logFileLocation
echo "Starting creation of resourceNo $resourceNo at $(date +"%Y-%m-%d %T")."
# Check if a new resource group should be created
if [ "$createRG" == "TRUE" ]; then
echo "Creating RG $newRgName$randomIdentifier at $(date +"%Y-%m-%d %T").">>$logFileLocation
az group create --location $location --name $newRgName$randomIdentifier >>$logFileLocation
existingRgName=$newRgName$randomIdentifier
echo " RG $newRgName$randomIdentifier creation complete"
fi
# Check if a new virtual network should be created, then create the VM
if [ "$createVnet" == "TRUE" ]; then
echo "Creating VNet $vnetName$randomIdentifier in RG $existingRgName at $(date +"%Y-%m-%d %T").">>$logFileLocation
az network vnet create \
--name $vnetName$randomIdentifier \
--resource-group $existingRgName \
--address-prefix $vnetAddressPrefix \
--subnet-name $subnetName$randomIdentifier \
--subnet-prefixes $subnetAddressPrefixes >>$logFileLocation
echo " VNet $vnetName$randomIdentifier creation complete"
echo "Creating VM $vmName$randomIdentifier in Vnet $vnetName$randomIdentifier in RG $existingRgName at $(date +"%Y-%m-%d %T").">>$logFileLocation
az vm create \
--resource-group $existingRgName \
--name $vmName$randomIdentifier \
--image $vmImage \
--vnet-name $vnetName$randomIdentifier \
--subnet $subnetName$randomIdentifier \
--public-ip-sku $publicIpSku \
--generate-ssh-keys >>$logFileLocation
echo " VM $vmName$randomIdentifier creation complete"
else
echo "Creating VM $vmName$randomIdentifier in RG $existingRgName at $(date +"%Y-%m-%d %T").">>$logFileLocation
az vm create \
--resource-group $existingRgName \
--name $vmName$randomIdentifier \
--image $vmImage \
--public-ip-sku $publicIpSku \
--admin-username $adminUser\
--admin-password $adminPassword$randomIdentifier >>$logFileLocation
echo " VM $vmName$randomIdentifier creation complete"
fi
# skip the header line
done < <(tail -n +2 $csvFileLocation)
# Clear the console (optional) and display the log file
# clear
cat $logFileLocation
In your console output, are you missing the last row in your CSV file? This can be caused by a missing line continuation character after the last line. Add a blank line at the end of your CSV file to fix the issue.
# Create Azure resources
# Create the log file
"CREATE AZURE RESOURCES." | Out-File -FilePath $logFileLocation
# Loop through each CSV row
foreach ($row in $data) {
$resourceNo = $row.resourceNo
$location = $row.location
$createRG = $row.createRG
$existingRgName = $row.existingRgName
$createVnet = $row.createVnet
$vnetAddressPrefix = $row.vnetAddressPrefix
$subnetAddressPrefixes = $row.subnetAddressPrefixes
$vmImage = $row.vmImage
$publicIpSku = $row.publicIpSku
$adminUser = $row.adminUser
# Generate a random ID
$randomIdentifier = (New-Guid).ToString().Substring(0,8)
# Log resource number, random ID and display start time
"" | Out-File -FilePath $logFileLocation -Append
"resourceNo = $resourceNo" | Out-File -FilePath $logFileLocation -Append
"randomIdentifier = $randomIdentifier" | Out-File -FilePath $logFileLocation -Append
Write-Host "Starting creation of resourceNo $resourceNo at $(Get-Date -format 'u')."
# Check if a new resource group should be created
if ($createRG -eq "TRUE") {
"Creating RG $newRgName$randomIdentifier at $(Get-Date -format 'u')." | Out-File -FilePath $logFileLocation -Append
az group create --location $location --name $newRgName$randomIdentifier | Out-File -FilePath $logFileLocation -Append
$existingRgName = "$newRgName$randomIdentifier"
Write-Host " RG $newRgName$randomIdentifier creation complete."
}
# Check if a new virtual network should be created, then create the VM
if ($createVnet -eq "TRUE") {
"Creating VNet $vnetName$randomIdentifier in RG $existingRgName at $(Get-Date -format 'u')." | Out-File -FilePath $logFileLocation -Append
az network vnet create `
--name $vnetName$randomIdentifier `
--resource-group $existingRgName `
--address-prefix $vnetAddressPrefix `
--subnet-name $subnetName$randomIdentifier `
--subnet-prefixes $subnetAddressPrefixes | Out-File -FilePath $logFileLocation -Append
Write-Host " VNet $vnetName$randomIdentifier creation complete."
"Creating VM $vmName$randomIdentifier in Vnet $vnetName$randomIdentifier in RG $existingRgName at $(Get-Date -format 'u')." | Out-File -FilePath $logFileLocation -Append
az vm create `
--resource-group $existingRgName `
--name $vmName$randomIdentifier `
--image $vmImage `
--vnet-name $vnetName$randomIdentifier `
--subnet $subnetName$randomIdentifier `
--public-ip-sku $publicIpSku `
--generate-ssh-keys | Out-File -FilePath $logFileLocation -Append
Write-Host " VM $vmName$randomIdentifier creation complete."
} else {
"Creating VM $vmName$randomIdentifier in RG $existingRgName at $(Get-Date -format 'u')." | Out-File -FilePath $logFileLocation -Append
az vm create `
--resource-group $existingRgName `
--name $vmName$randomIdentifier `
--image $vmImage `
--public-ip-sku $publicIpSku `
--admin-username $adminUser `
--admin-password $adminPassword$randomIdentifier | Out-File -FilePath $logFileLocation -Append
Write-Host " VM $vmName$randomIdentifier creation complete."
}
}
# Clear the console (optional) and display the log file
# Clear-Host
Get-Content -Path $logFileLocation
Console output before log file read:
Starting creation of resourceNo 1 at YYYY-MM-DD HH:MM:SS.
RG msdocs-rg-00000001 creation complete
VNet msdocs-vnet-00000001 creation complete
VM msdocs-vm-00000001 creation complete
Starting creation of resourceNo 2 at YYYY-MM-DD HH:MM:SS.
RG msdocs-rg-00000002 creation complete
VM msdocs-vm-00000002 creation complete
Starting creation of resourceNo 3 at YYYY-MM-DD HH:MM:SS.
VM msdocs-vm-00000003 creation complete
Your log file contents should look similar to this:
Starting creation of resourceNo 1 at YYYY-MM-DD HH:MM:SS.
Creating RG msdocs-rg-00000001 at YYYY-MM-DD HH:MM:SS.
{
Resource group create output
}
Creating VNet msdocs-vnet-00000001 in RG msdocs-rg-000000001 at YYYY-MM-DD HH:MM:SS.
{
VNet create output
}
Creating VM msdocs-vm-00000001 in RG msdocs-rg-00000001 at YYYY-MM-DD HH:MM:SS.
{
VM create output
}
Starting creation of resourceNo 2 at YYYY-MM-DD HH:MM:SS.
Creating RG msdocs-rg-00000002 at YYYY-MM-DD HH:MM:SS.
{
Resource group create output
}
Creating VM msdocs-vm-00000002 in RG msdocs-rg-00000002 at YYYY-MM-DD HH:MM:SS.
{
VM create output
}
Starting creation of resourceNo 3 at YYYY-MM-DD HH:MM:SS.
Creating msdocs-vm-00000003 creation complete
{
VM create output
}
Troubleshooting
In Bash, the "Create Azure resources" step stops after step 1
Bash is case sensitive. The word true does not equal TRUE. Also greater than is -gt, not >, and equals is ==, not =. Make sure you do not have a typographical error, or leading/trailing spaces in your CSV column values.
Variable values are not changing with each loop
This is often caused by extra spaces in the CSV file. A line in a CSV file will look something like this: column1,column2,column3 or column1,,column3, but by habit it is easy to create a test file that contains a space after each comma like column1, column2, column3. When you have a leading or trailing space in your CSV, the column value is actually <space>columnValue. The script logic if [ "$columnName" = "columnValue" ] returns "false". Remove all leading and trailing spaces in your CSV rows to fix the issue.
Invalid CIDR notation
You receive an InvalidCIDRNotation error when you pass an incorrect address prefix to az network vnet create. This can be challenging when visually, the address prefix looks correct when returned in an echo statement. To troubleshoot the actual value being read from the CSV, try this script:
while IFS=, read -r resourceNo location createRG existingRgName createVnet vnetAddressPrefix subnetAddressPrefixes vmImage publicIpSku adminUser
do
echo "resourceNo = $resourceNo"
if [ "$createVnet" == "TRUE" ]; then
startTest="abc"
endTest="xyz"
echo $startTest$vnetAddressPrefix$endTest
fi
done < <(tail -n +2 $setupFileLocation)
If your results look like xzy10.0.0.0 and not the expected abc10.0.0.0/24xyz, there might be a hidden character or extra comma lurking in your CSV file. Add a test column with the same prefix value, rearrange your CSV columns, and copy/paste your CSV contents in/out of a simple Notepad editor. In writing this article, the rearrangement of the CSV columns finally fixed the error.
Arguments are expected or required
You receive this error when you have not supplied a required parameter or there is a typographical error that causes the Azure CLI to incorrectly parse the reference command. When working with a script, you also receive this error when one of more of the following is true:
There is a missing or incorrect line continuation character.
There are trailing spaces on the right side of a line continuation character.
Your variable name contains a special character, such as a dash (-).
InvalidTemplateDeployment
When you try to create an Azure resource in a location that does not offer that resource you receive an error similar to the following: "Following SKUs have failed for Capacity Restrictions: Standard_DS1_v2' is currently not available in location 'westus'."
Here's the full error example:
{"error":{"code":"InvalidTemplateDeployment","message":"The template deployment 'vm_deploy_<32 character ID>'
is not valid according to the validation procedure. The tracking id is '<36 character ID>'.
See inner errors for details.","details":[{"code":"SkuNotAvailable","message":"The requested VM size for resource
'Following SKUs have failed for Capacity Restrictions: Standard_DS1_v2' is currently not available
in location '<your specified location>'. Please try another size or deploy to a different location
or different zone. See https://aka.ms/azureskunotavailable for details."}]}}
To correct the error, either change the location or select a different parameter value that is offered for your desired location.
The source for this content can be found on GitHub, where you can also create and review issues and pull requests. For more information, see our contributor guide.
Azure CLI feedback
Azure CLI is an open source project. Select a link to provide feedback: