error in custom template deployment for cloudera hdp on azure

Question

Dear friends,

I try to install the open-source Cloudera hdp on azure. The ones existing in azure marketplace do not work, see below
https://learn.microsoft.com/en-us/answers/questions/1043115/connection-to-azure-vm-closed-system-is-booting-up.html?childToView=1047166#comment-1047166

I try to install it via the custom template deployment appended below, but receive the following error

{"code":"MultipleErrorsOccurred","details":[{"code":"InvalidTemplate","target":"/subscriptions/abdafe74-cd54-4de5-947a-70e1b4488aaf/resourceGroups/HDP_on_Azure2/providers/Microsoft.Resources/deployments/data-node","message":"Deployment template parse failed: 'Required property '$schema' not found in JSON. Path '', line 1, position 92.'."},{"code":"InvalidTemplate","target":"/subscriptions/abdafe74-cd54-4de5-947a-70e1b4488aaf/resourceGroups/HDP_on_Azure2/providers/Microsoft.Resources/deployments/shared-resources","message":"Deployment template validation failed: 'The template parameters 'networkSpec, masterNodeASName, dataNodeASName' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time. The only supported parameters for this template are 'vmSize, company, jobRole, vmCount, vmImage, lastName, firstName, cmPassword, cmUsername, subnetName, tshirtSize, jobFunction, emailAddress, subnetPrefix, addressPrefix, adminPassword, adminUsername, businessPhone, dnsNamePrefix, numberOfDataNodes, vnetNewOrExisting, virtualNetworkName, masterNodeIPAddress, publicIPAddressName, virtualNetworkRGName, masterStorageAccountType, workerStorageAccountType, dataNodeIPOffSetFromMaster'. Please see https://aka.ms/arm-deploy/#parameter-file for usage details.'.","additionalInfo":[{"type":"TemplateViolation","info":{"lineNumber":0,"linePosition":0,"path":""}}]},{"code":"InvalidTemplate","target":"/subscriptions/abdafe74-cd54-4de5-947a-70e1b4488aaf/resourceGroups/HDP_on_Azure2/providers/Microsoft.Resources/deployments/setup-cloudera","message":"Deployment template validation failed: 'The template parameters 'clusterSpec, installCDH' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time. The only supported parameters for this template are 'fqdn, vmSize, vmSpec, company, jobRole, vmCount, vmImage, lastName, location, firstName, cmPassword, cmUsername, scriptsUri, subnetName, tshirtSize, jobFunction, networkSpec, emailAddress, subnetPrefix, addressPrefix, adminPassword, adminUsername, businessPhone, dnsNamePrefix, numberOfDataNodes, vnetNewOrExisting, virtualNetworkName, masterNodeIPAddress, publicIPAddressName, virtualNetworkRGName, masterStorageAccountType, workerStorageAccountType, dataNodeIPOffSetFromMaster'. Please see https://aka.ms/arm-deploy/#parameter-file for usage details.'.","additionalInfo":[{"type":"TemplateViolation","info":{"lineNumber":0,"linePosition":0,"path":""}}]}],"message":"Multiple error occurred: BadRequest,BadRequest,BadRequest. Please see details."}

{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json",
"contentVersion": "1.0.0.1",
"parameters": {
"adminUsername": {
"type": "string",
"defaultValue": "azureuser",
"metadata": {
"description": "Admin user name for the VMs; Get-AzMarketplaceTerms -Publisher cloudera -Product cloudera-centos-6 -Name cloudera-centos-6 | Set-AzMarketplaceTerms -Accept"
}
},
"adminPassword": {
"type": "securestring",
"defaultValue": "*****",
"metadata": {
"description": "Admin password for the VMs (A mix of upper and lower-case characters, digits and symbols)"
}
},
"cmUsername": {
"type": "string",
"defaultValue": "cmadmin",
"metadata": {
"description": "User name for the Cloudera Manager"
}
},
"cmPassword": {
"type": "securestring",
"defaultValue": "*****",
"metadata": {
"description": "password for the Cloudera Manager (A mix of upper and lower-case characters, digits and symbols)"
}
},
"dnsNamePrefix": {
"type": "string",
"defaultValue": "azuredevcluster",
"metadata": {
"description": "Unique public DNS name where the VMs will be exposed"
}
},
"masterStorageAccountType": {
"type": "string",
"defaultValue": "Premium_LRS",
"metadata": {
"description": "The type of the Storage Account to be created for master nodes (defaults to Premium_LRS)"
}
},
"workerStorageAccountType": {
"type": "string",
"defaultValue": "Standard_LRS",
"metadata": {
"description": "The type of the Storage Account to be created for worker nodes (defaults to Standard_LRS)"
}
},
"virtualNetworkName": {
"type": "string",
"defaultValue": "clouderavnet",
"metadata": {
"description": "The name of the virtual network provisioned for the deployment"
}
},
"vnetNewOrExisting": {
"type": "string",
"defaultValue": "new",
"allowedValues": [
"new",
"existing"
],
"metadata": {
"description": "Indicator for new or exiting Virtual Network"
}
},
"virtualNetworkRGName": {
"type": "string",
"defaultValue": "AzureDevCluster",
"metadata": {
"description": "Resource Group Name for Vnet. For new VNet leave it empty, otherwise type in existing resource group name"
}
},
"subnetName": {
"type": "string",
"defaultValue": "clouderasubnet",
"metadata": {
"description": "Subnet name for the virtual network where resources will be provisioned"
}
},
"addressPrefix": {
"type": "string",
"defaultValue": "10.3.0.0/16",
"metadata": {
"description": "Virtual Network address CIDR"
}
},
"subnetPrefix": {
"type": "string",
"defaultValue": "10.3.0.0/16",
"metadata": {
"description": "CIDR for the subnet where VMs will be placed"
}
},
"publicIPAddressName": {
"type": "string",
"defaultValue": "MyPIP",
"metadata": {
"description": "Public IP address name"
}
},
"masterNodeIPAddress": {
"type": "string",
"defaultValue": "10.3.0.4",
"metadata": {
"description": "IP address for the first master"
}
},
"dataNodeIPOffSetFromMaster": {
"type": "int",
"defaultValue": 5,
"metadata": {
"description": "IP address from the master node, for example if the first master is 10.1.1.1, then the first dataNode would be 10.1.1.11"
}
},
"tshirtSize": {
"type": "string",
"defaultValue": "Eval",
"allowedValues": [
"Eval",
"Prod"
],
"metadata": {
"description": "T-shirt size of the Cloudera cluster (Eval, Prod)"
}
},
"vmCount": {
"type": "int",
"defaultValue": 1,
"metadata": {
"description": "Number of VMs"
}
},
"numberOfDataNodes": {
"type": "int",
"defaultValue": 1,
"metadata": {
"description": "Number of data nodes for Prod (defaults to 3)"
}
},
"vmSize": {
"type": "string",
"defaultValue": "Standard_A4_v2",
"metadata": {
"description": "The size of the VMs deployed in the cluster (defaults to Standard_A4_v2)"
}
},
"vmImage": {
"type": "string",
"defaultValue": "ClouderaCentOS6_7",
"metadata": {
"description": "The OS VM Image (defaults to ClouderaCentOS6_7)"
},
"allowedValues": [
"ClouderaCentOS6_7",
"ClouderaCentOS6_8"
]
},
"company": {
"type": "string",
"defaultValue": "Company",
"metadata": {
"description": "Your Company"
}
},
"emailAddress": {
"type": "string",
"defaultValue": "******@yourcompany.com",
"metadata": {
"description": "your email"
}
},
"businessPhone": {
"type": "string",
"defaultValue": "123456789",
"metadata": {
"description": "your business phone number"
}
},
"firstName": {
"type": "string",
"defaultValue": "FirstName",
"metadata": {
"description": "Your FirstName"
}
},
"lastName": {
"type": "string",
"defaultValue": "LastName",
"metadata": {
"description": "Your LastName"
}
},
"jobRole": {
"type": "string",
"defaultValue": "Administrator",
"metadata": {
"description": "Job Role"
},
"allowedValues": [
"Administrator",
"Analyst (Industry/Finance)",
"Analyst (Technical)",
"Architect",
"C level",
"Consultant",
"Developer",
"Director",
"Engineer",
"Individual contributor",
"Manager",
"Partner",
"Student",
"VP"
]
},
"jobFunction": {
"type": "string",
"defaultValue": "Analytics / Business Intelligence",
"metadata": {
"description": "Job Function"
},
"allowedValues": [
"Analytics / Business Intelligence",
"Big Data / Architecture",
"Data Warehouse / Management",
"Data Science",
"Database",
"ETL / ELT",
"Infrastructure",
"IT",
"Marketing / Sales",
"Networking",
"Manager",
"Security / Compliance",
"Storage"
]
}
},
"variables": {
"installCDH": "True",
"addressPrefix": "[parameters('addressPrefix')]",
"subnetPrefix": "[parameters('subnetPrefix')]",
"scriptsUri": "https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/cloudera-on-centos",
"share-resourcesUri": "[concat(variables('scriptsUri'), '/shared-resources-', parameters('vnetNewOrExisting'), '-vnet.json')]",
"data-nodeUri": "[concat(variables('scriptsUri'), '/data-node-', toLower(subString(parameters('vmSize'), 0, 13)), '.json')]",
"sa": "[parameters('masterNodeIPAddress')]",
"ipOctet01": "[concat(split(variables('sa'), '.')[0], '.', split(variables('sa'), '.')[1], '.')]",
"ipOctet2": "[int(split(variables('sa'), '.')[2])]",
"ipOctet3": "[int(split(variables('sa'), '.')[3])]",
"datanodeIpOctet3": "[add(int(split(variables('sa'), '.')[3]), parameters('dataNodeIPOffSetFromMaster'))]",
"masterStorageAccount": {
"type": "[parameters('masterStorageAccountType')]"
},
"workerStorageAccount": {
"type": "[parameters('workerStorageAccountType')]"
},
"clusterTshirtSizeEval": {
"masterNodeCount": 1,
"dataNodeCount": "[parameters('numberOfDataNodes')]",
"highAvailability": "false"
},
"clusterTshirtSizeProd": {
"masterNodeCount": 1,
"dataNodeCount": "[parameters('numberOfDataNodes')]",
"highAvailability": "true"
},
"clusterSpec": "[variables(concat('clusterTshirtSize', parameters('tshirtSize')))]",
"vmImageInfo": {
"ClouderaCentOS6_7": {
"plan": {
"name": "cloudera-centos-6",
"publisher": "cloudera",
"product": "cloudera-centos-6"
},
"imageReference": {
"publisher": "cloudera",
"offer": "cloudera-centos-6",
"sku": "cloudera-centos-6",
"version": "latest"
}
},
"ClouderaCentOS6_8": {
"plan": {
"name": "6_8",
"publisher": "cloudera",
"product": "cloudera-centos-os"
},
"imageReference": {
"publisher": "cloudera",
"offer": "cloudera-centos-os",
"sku": "6_8",
"version": "latest"
}
},
"ClouderaCentOS6_7Germany": {
"plan": null,
"imageReference": {
"publisher": "cloudera",
"offer": "cloudera-centos-os",
"sku": "6_7",
"version": "latest"
}
}
},
"vmSpec": {
"vmSize": "[parameters('vmSize')]",
"imageInfo": "[variables('vmImageInfo')[parameters('vmImage')]]",
"adminUsername": "[parameters('adminUsername')]",
"adminPassword": "[parameters('adminPassword')]",
"masterNodeASName": "[concat(parameters('dnsNamePrefix'), '-mnAS')]",
"dataNodeASName": "[concat(parameters('dnsNamePrefix'), '-dnAS')]"
},
"networkSpec": {
"addressPrefix": "[variables('addressPrefix')]",
"subnetPrefix": "[variables('subnetPrefix')]",
"ipOctet01": "[variables('ipOctet01')]",
"ipOctet2": "[variables('ipOctet2')]",
"ipOctet3": "[variables('ipOctet3')]",
"datanodeIpOctet3": "[variables('datanodeIpOctet3')]",
"masterIP": "[variables('sa')]",
"workerIP": "[concat(variables('ipOctet01'), add(variables('ipOctet2'), div(variables('datanodeIpOctet3'), 255)), '.', mod(variables('datanodeIpOctet3'), 255))]",
"virtualNetworkName": "[parameters('virtualNetworkName')]",
"virtualNetworkRGName": "[parameters('virtualNetworkRGName')]",
"vnetNewOrExisting": "[parameters('vnetNewOrExisting')]",
"virtualNetworkSubnetName": "[parameters('subnetName')]"
},
"newVNetId": "[resourceId(concat('Microsoft.Network','/','virtualNetworks'),parameters('virtualNetworkName'))]",
"existingVNetId": "[resourceId(parameters('virtualNetworkRGName'),concat('Microsoft.Network','/','virtualNetworks'),parameters('virtualNetworkName'))]",
"VNetId": "[variables(concat(parameters('vnetNewOrExisting'),'VNetId'))]",
"vmName": "[concat(parameters('dnsNamePrefix'), '-mn')]",
"publicIPAddressName": "[concat(variables('vmName'), '-publicIP')]"
},
"resources": [
{
"name": "shared-resources",
"type": "Microsoft.Resources/deployments",
"apiVersion": "2018-09-01",
"properties": {
"mode": "Incremental",
"templateLink": {
"uri": "https://api.npoint.io/aaa3097edeece4795131"
},
"parameters": {
"networkSpec": {
"value": "[variables('networkSpec')]"
},
"masterNodeASName": {
"value": "[variables('vmSpec').masterNodeASName]"
},
"dataNodeASName": {
"value": "[variables('vmSpec').dataNodeASName]"
}
}
}
},
{
"name": "master-node",
"type": "Microsoft.Resources/deployments",
"apiVersion": "2018-09-01",
"dependsOn": [
"Microsoft.Resources/deployments/shared-resources"
],
"properties": {
"mode": "Incremental",
"templateLink": {
"uri": "https://api.npoint.io/01b5ff563820aefe52ac"
},
"parameters": {
"vnetID": {
"value": "[variables('VNetId')]"
},
"dnsNamePrefix": {
"value": "[parameters('dnsNamePrefix')]"
},
"scriptsUri": {
"value": "[variables('scriptsUri')]"
},
"storageAccount": {
"value": "[variables('masterStorageAccount')]"
},
"vmCount": {
"value": "[variables('clusterSpec').masterNodeCount]"
},
"vmSpec": {
"value": "[variables('vmSpec')]"
},
"networkSpec": {
"value": "[variables('networkSpec')]"
},
"clusterSpec": {
"value": "[variables('clusterSpec')]"
}
}
}
},
{
"name": "data-node",
"type": "Microsoft.Resources/deployments",
"apiVersion": "2018-09-01",
"dependsOn": [
"Microsoft.Resources/deployments/shared-resources"
],
"properties": {
"mode": "Incremental",
"templateLink": {
"uri": "https://api.npoint.io/478453f61c2db5d4e50d"
},
"parameters": {
"vnetID": {
"value": "[variables('VNetId')]"
},
"dnsNamePrefix": {
"value": "[parameters('dnsNamePrefix')]"
},
"scriptsUri": {
"value": "[variables('scriptsUri')]"
},
"vmCount": {
"value": "[variables('clusterSpec').dataNodeCount]"
},
"vmSpec": {
"value": "[variables('vmSpec')]"
},
"networkSpec": {
"value": "[variables('networkSpec')]"
},
"clusterSpec": {
"value": "[variables('clusterSpec')]"
}
}
}
},
{
"name": "setup-cloudera",
"type": "Microsoft.Resources/deployments",
"apiVersion": "2018-09-01",
"dependsOn": [
"Microsoft.Resources/deployments/data-node",
"Microsoft.Resources/deployments/master-node"
],
"properties": {
"mode": "Incremental",
"templateLink": {
"uri": "https://api.npoint.io/840c64149b4b33565b0d"
},
"parameters": {
"dnsNamePrefix": {
"value": "[parameters('dnsNamePrefix')]"
},
"scriptsUri": {
"value": "[variables('scriptsUri')]"
},
"vmSpec": {
"value": "[variables('vmSpec')]"
},
"networkSpec": {
"value": "[variables('networkSpec')]"
},
"clusterSpec": {
"value": "[variables('clusterSpec')]"
},
"fqdn": {
"value": "[concat(parameters('dnsNamePrefix'), '-mn0.', resourceGroup().location, '.cloudapp.azure.com')]"
},
"cmUsername": {
"value": "[parameters('cmUsername')]"
},
"cmPassword": {
"value": "[parameters('cmPassword')]"
},
"company": {
"value": "[parameters('Company')]"
},
"emailAddress": {
"value": "[parameters('emailAddress')]"
},
"businessPhone": {
"value": "[parameters('businessPhone')]"
},
"firstName": {
"value": "[parameters('firstName')]"
},
"lastName": {
"value": "[parameters('lastName')]"
},
"jobRole": {
"value": "[parameters('jobRole')]"
},
"jobFunction": {
"value": "[parameters('jobFunction')]"
},
"installCDH": {
"value": "[variables('installCDH')]"
}
}
}
}
]
}

Answer 1

Hello @DiscouMap

Thank you for your post on this community space.

I have read the link gathered on this post description and this was similar issue from the one described at the very beginning.

For instance, the error observed is showing that the original provision template does not contain some parameters used as shown below:

The template parameters 'networkSpec, masterNodeASName, dataNodeASName' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time. The only supported parameters for this template are 'vmSize, company, jobRole, vmCount, vmImage, lastName, firstName, cmPassword, cmUsername, subnetName, tshirtSize, jobFunction, emailAddress, subnetPrefix, addressPrefix, adminPassword, adminUsername, businessPhone, dnsNamePrefix, numberOfDataNodes, vnetNewOrExisting, virtualNetworkName, masterNodeIPAddress, publicIPAddressName, virtualNetworkRGName, masterStorageAccountType, workerStorageAccountType, dataNodeIPOffSetFromMaste

Just in case, you might check if this is registered "Microsoft.Solutions resource provider" and try it again to see if works or not.

Furthermore,I would like to share the following articles down below:

https://learn.microsoft.com/en-us/azure/azure-resource-manager/managed-applications/overview

https://learn.microsoft.com/en-us/azure/marketplace/azure-vm-use-own-image

https://learn.microsoft.com/en-us/azure/marketplace/marketplace-virtual-machines

So I would say that you need support assistance from this vendor in case you do not get this working as you were expecting.

Looking forward to your feedback,

Cheers,

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Answer 2

Dear all,

I think, I managed to solve most of the problems, there is still one error in shared-resources, which I can not understand it.

{"code":"MultipleErrorsOccurred","details":[{"code":"InvalidTemplate","target":"/subscriptions/abdafe74-cd54-4de5-947a-70e1b4488aaf/resourceGroups/HDP_on_Azure2/providers/Microsoft.Resources/deployments/shared-resources","message":"Deployment template parse failed: 'Required property 'type' not found in JSON. Path 'parameters.networkSpec', line 1, position 1775.'."},{"code":"InvalidTemplate","target":"/subscriptions/abdafe74-cd54-4de5-947a-70e1b4488aaf/resourceGroups/HDP_on_Azure2/providers/Microsoft.Resources/deployments/setup-cloudera","message":"Deployment template validation failed: 'The template parameters 'clusterSpec, installCDH' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time. The only supported parameters for this template are 'fqdn, vmSize, vmSpec, company, jobRole, vmCount, vmImage, lastName, location, firstName, cmPassword, cmUsername, scriptsUri, subnetName, tshirtSize, jobFunction, networkSpec, emailAddress, subnetPrefix, addressPrefix, adminPassword, adminUsername, businessPhone, dnsNamePrefix, numberOfDataNodes, vnetNewOrExisting, virtualNetworkName, masterNodeIPAddress, publicIPAddressName, virtualNetworkRGName, masterStorageAccountType, workerStorageAccountType, dataNodeIPOffSetFromMaster'. Please see https://aka.ms/arm-deploy/#parameter-file for usage details.'.","additionalInfo":[{"type":"TemplateViolation","info":{"lineNumber":0,"linePosition":0,"path":""}}]}],"message":"Multiple error occurred: BadRequest,BadRequest. Please see details."}

{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json",
"contentVersion": "1.0.0.1",
"parameters": {
"resourceAPIVersion": {
"type": "string"
},
"networkSpec": {
"type": "object"
},
"masterNodeASName": {
"type": "string"
},
"dataNodeASName": {
"type": "string"
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Location for all resources."
}
}
},
"variables": {},
"resources": [
{
"apiVersion": "2017-06-01",
"type": "Microsoft.Compute/availabilitySets",
"name": "[parameters('masterNodeASName')]",
"location": "East US",
"sku": {
"name": "Aligned"
},
"properties": {
"platformFaultDomainCount": 2,
"platformUpdateDomainCount": 5
}
},
{
"apiVersion": "2017-06-01",
"type": "Microsoft.Compute/availabilitySets",
"name": "[parameters('dataNodeASName')]",
"location": "East US",
"sku": {
"name": "Aligned"
},
"properties": {
"platformFaultDomainCount": 2,
"platformUpdateDomainCount": 5
}
},
{
"apiVersion": "2017-06-01",
"type": "Microsoft.Network/networkSecurityGroups",
"name": "[concat(parameters('networkSpec').virtualNetworkName, '-sg')]",
"location": "[parameters('location')]",
"properties": {
"securityRules": [
{
"name": "SSH",
"properties": {
"description": "Allows SSH traffic",
"protocol": "Tcp",
"sourcePortRange": "",
"destinationPortRange": "22",
"sourceAddressPrefix": "",
"destinationAddressPrefix": "*",
"access": "Allow",
"priority": 100,
"direction": "Inbound"
}
}
]
}
},
{
"apiVersion": "2017-06-01",
"type": "Microsoft.Network/virtualNetworks",
"name": "[parameters('networkSpec').virtualNetworkName]",
"dependsOn": [
"[concat('Microsoft.Network/networkSecurityGroups/', variables('networkSpec').virtualNetworkName, '-sg')]"
],
"location": "East US",
"properties": {
"addressSpace": {
"addressPrefixes": [
"[variables('networkSpec').addressPrefix]"
]
},
"subnets": [
{
"name": "[variables('networkSpec').virtualNetworkSubnetName]",
"properties": {
"addressPrefix": "[variables('networkSpec').subnetPrefix]",
"networkSecurityGroup": {
"id": "[resourceId('Microsoft.Network/networkSecurityGroups', concat(variables('networkSpec').virtualNetworkName, '-sg'))]"
}
}
}
]
}
}
],
"parameters": {
"resourceAPIVersion": {
"value": "[variables('resourceAPIVersion')]"
},
"networkSpec": {
"value": "[variables('networkSpec')]"
},
"masterNodeASName": {
"value": "[variables('vmSpec').masterNodeASName]"
},
"dataNodeASName": {
"value": "[variables('vmSpec').dataNodeASName]"
}

}  

}

Answer 3

Hello @DiscouMap

Thank you for your answer.

I am glad to hear that you were able to solve most of the deployment errors. I was thinking of checking the resource provider registration status at the subscription level.

You can either double-check the API version for each resource provider or if the resource provider required is registered or not.

Looking forward to your feedback,

Cheers,

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Answer 4

I m still trying to understand this error. Everything sounds fine, and there is a type associated with 'parameters.networkSpec'

"Deployment template parse failed: 'Required property 'type' not found in JSON. Path 'parameters.networkSpec', line 1, position 1775.