Powershell: Deserialization error on Spark pool creation

Question

Powershell: Deserialization error on Spark pool creation

Razeen Hendricks 20

Hi,

I'm trying to create a spark pool via Powershell but get the following error:

Operation returned an invalid status code 'BadRequest' Spark pool request validation failed. SparkPoolConfigContent Deserialization Failed. Content should be deserialied to SparkConfigurationInfoObject.

The following is the script I'm using:

$config = Get-AzSynapseSparkConfiguration -WorkspaceName $WORKSPACE -Name $CONFIG_NAME  

New-AzSynapseSparkPool -ResourceGroupName $RESOURCE_GROUP_NAME -WorkspaceName $WORKSPACE -Name $SPARK_POOL_NAME `
     -EnableDynamicExecutorAllocation -MinExecutorCount $MIN_EXECUTOR_COUNT -MaxExecutorCount $MAX_EXECUTOR_COUNT `
     -SparkVersion $SPARK_VERSION -NodeSize $NODE_SIZE -AutoScaleMinNodeCount $MIN_NODE_COUNT -AutoScaleMaxNodeCount $MAX_NODE_COUNT `
     -EnableAutoPause -AutoPauseDelayInMinute $PAUSE_DELAY_MINS -SparkConfiguration $config

The documentation I've followed is: https://learn.microsoft.com/en-us/powershell/module/az.synapse/new-azsynapsesparkpool?view=azps-10.4.1#example-6

Why is the error being encountered? How could I get this to work?

PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2023-12-05T05:23:05.2566667+00:00

@Razeen Hendricks - Thanks for the question and using MS Q&A platform.

The error message suggests that there is an issue with the deserialization of the SparkPoolConfigContent. This could be due to an issue with the format of the configuration content or an issue with the PowerShell cmdlet itself.

To troubleshoot this issue, check the format of the configuration content. Make sure that it is in the correct format and that all required fields are present.

In case, if you are unable to resolve the issue, kindly share the complete script along with the parameters passed for the above powershell script.

Looking forward for your response.
Razeen Hendricks 20 Reputation points

2023-12-05T15:19:08.65+00:00
Thanks for the reply @PRADEEPCHEEKATLA

The configuration was uploaded via the Synapse WS UI, so whatever format it's in there, that's controlled by Microsoft. I tried searching for SparkConfigurationInfoObject, but I get no results. I can't find any other relevant info on the web about this. Microsoft documentation is rarely detailed enough, especially with troubleshooting.

If the documentation from the link I used is accurate, this should work?

You've indicated that I should check that it's in the correct format. What exactly is the correct format? Can you please provide more info on what should be passed, and how that parameter can be produced? I need your guidance please.

The code I provided to you is all that I'm executing, I've only left out the parameter declarations, which is as follows:

Param( [string] [Parameter(Mandatory=$true)] $RESOURCE_GROUP_NAME, [string] [Parameter(Mandatory=$true)] $WORKSPACE, [string] [Parameter(Mandatory=$true)] $SPARK_POOL_NAME, [string] [Parameter(Mandatory=$true)] $SPARK_VERSION, [string] [Parameter(Mandatory=$true)] $NODE_SIZE, [int] [Parameter(Mandatory=$true)] $MIN_NODE_COUNT, [int] [Parameter(Mandatory=$true)] $MAX_NODE_COUNT, [int] [Parameter(Mandatory=$true)] $MIN_EXECUTOR_COUNT, [int] [Parameter(Mandatory=$true)] $MAX_EXECUTOR_COUNT, [int] [Parameter(Mandatory=$true)] $PAUSE_DELAY_MINS, [string] [Parameter(Mandatory=$true)] $CONFIG_NAME )

-RESOURCE_GROUP_NAME rg-001 -WORKSPACE syn-001 -SPARK_POOL_NAME sparkpool01 -SPARK_VERSION 3.3 -NODE_SIZE Large -MIN_NODE_COUNT 3 -MAX_NODE_COUNT 10 -MIN_EXECUTOR_COUNT 1 -MAX_EXECUTOR_COUNT 4 -PAUSE_DELAY_MINS 15 -CONFIG_NAME config
PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2023-12-07T07:13:42.68+00:00
@Razeen Hendricks - I apologize for the confusion. The correct format for the SparkPoolConfigContent parameter is a JSON string that contains the Spark configuration settings. You can obtain this JSON string by exporting the Spark configuration settings from the Synapse workspace UI.

To export the Spark configuration settings, follow these steps:

Open the Synapse workspace in the Azure portal.

Click on the "Manage" tab in the left-hand menu.

Click on "Apache Spark pools" and select the Spark pool you want to export the configuration for.

Click on the "Export configuration" button at the top of the page.

Save the exported JSON file to your local machine.

Once you have the JSON file, you can read it into a PowerShell variable and pass it to the New-AzSynapseSparkPool cmdlet. Here's an example:

$config = Get-Content -Path "C:\path\to\config.json" -Raw New-AzSynapseSparkPool -ResourceGroupName $RESOURCE_GROUP_NAME -WorkspaceName $WORKSPACE -Name $SPARK_POOL_NAME ` -EnableDynamicExecutorAllocation -MinExecutorCount $MIN_EXECUTOR_COUNT -MaxExecutorCount $MAX_EXECUTOR_COUNT ` -SparkVersion $SPARK_VERSION -NodeSize $NODE_SIZE -AutoScaleMinNodeCount $MIN_NODE_COUNT -AutoScaleMaxNodeCount $MAX_NODE_COUNT ` -EnableAutoPause -AutoPauseDelayInMinute $PAUSE_DELAY_MINS -SparkPoolConfigContent $config

For more details, refer to https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-azure-create-spark-configuration#import-and-export-an-apache-spark-configuration

I hope this helps! Let me know if you have any other questions.
PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2023-12-13T02:47:22.92+00:00

@Razeen Hendricks - Did you get a chance to accept the summarized answer?

Just checking in to see if the below answer helped. If this answers your query, do click "Accept the answer” for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Answer accepted by question author

1 additional answer

Your answer

PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2023-12-05T05:23:05.2566667+00:00

@Razeen Hendricks - Thanks for the question and using MS Q&A platform.

The error message suggests that there is an issue with the deserialization of the SparkPoolConfigContent. This could be due to an issue with the format of the configuration content or an issue with the PowerShell cmdlet itself.

To troubleshoot this issue, check the format of the configuration content. Make sure that it is in the correct format and that all required fields are present.

In case, if you are unable to resolve the issue, kindly share the complete script along with the parameters passed for the above powershell script.

Looking forward for your response.
Razeen Hendricks 20 Reputation points

2023-12-05T15:19:08.65+00:00

Thanks for the reply @PRADEEPCHEEKATLA

The configuration was uploaded via the Synapse WS UI, so whatever format it's in there, that's controlled by Microsoft. I tried searching for SparkConfigurationInfoObject, but I get no results. I can't find any other relevant info on the web about this. Microsoft documentation is rarely detailed enough, especially with troubleshooting.

If the documentation from the link I used is accurate, this should work?

You've indicated that I should check that it's in the correct format. What exactly is the correct format? Can you please provide more info on what should be passed, and how that parameter can be produced? I need your guidance please.

The code I provided to you is all that I'm executing, I've only left out the parameter declarations, which is as follows:

Param( [string] [Parameter(Mandatory=$true)] $RESOURCE_GROUP_NAME, [string] [Parameter(Mandatory=$true)] $WORKSPACE, [string] [Parameter(Mandatory=$true)] $SPARK_POOL_NAME, [string] [Parameter(Mandatory=$true)] $SPARK_VERSION, [string] [Parameter(Mandatory=$true)] $NODE_SIZE, [int] [Parameter(Mandatory=$true)] $MIN_NODE_COUNT, [int] [Parameter(Mandatory=$true)] $MAX_NODE_COUNT, [int] [Parameter(Mandatory=$true)] $MIN_EXECUTOR_COUNT, [int] [Parameter(Mandatory=$true)] $MAX_EXECUTOR_COUNT, [int] [Parameter(Mandatory=$true)] $PAUSE_DELAY_MINS, [string] [Parameter(Mandatory=$true)] $CONFIG_NAME )

-RESOURCE_GROUP_NAME rg-001 -WORKSPACE syn-001 -SPARK_POOL_NAME sparkpool01 -SPARK_VERSION 3.3 -NODE_SIZE Large -MIN_NODE_COUNT 3 -MAX_NODE_COUNT 10 -MIN_EXECUTOR_COUNT 1 -MAX_EXECUTOR_COUNT 4 -PAUSE_DELAY_MINS 15 -CONFIG_NAME config
PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2023-12-07T07:13:42.68+00:00

@Razeen Hendricks - I apologize for the confusion. The correct format for the SparkPoolConfigContent parameter is a JSON string that contains the Spark configuration settings. You can obtain this JSON string by exporting the Spark configuration settings from the Synapse workspace UI.

To export the Spark configuration settings, follow these steps:

Open the Synapse workspace in the Azure portal.

Click on the "Manage" tab in the left-hand menu.

Click on "Apache Spark pools" and select the Spark pool you want to export the configuration for.

Click on the "Export configuration" button at the top of the page.

Save the exported JSON file to your local machine.

Once you have the JSON file, you can read it into a PowerShell variable and pass it to the New-AzSynapseSparkPool cmdlet. Here's an example:

$config = Get-Content -Path "C:\path\to\config.json" -Raw New-AzSynapseSparkPool -ResourceGroupName $RESOURCE_GROUP_NAME -WorkspaceName $WORKSPACE -Name $SPARK_POOL_NAME ` -EnableDynamicExecutorAllocation -MinExecutorCount $MIN_EXECUTOR_COUNT -MaxExecutorCount $MAX_EXECUTOR_COUNT ` -SparkVersion $SPARK_VERSION -NodeSize $NODE_SIZE -AutoScaleMinNodeCount $MIN_NODE_COUNT -AutoScaleMaxNodeCount $MAX_NODE_COUNT ` -EnableAutoPause -AutoPauseDelayInMinute $PAUSE_DELAY_MINS -SparkPoolConfigContent $config

For more details, refer to https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-azure-create-spark-configuration#import-and-export-an-apache-spark-configuration

I hope this helps! Let me know if you have any other questions.
PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2023-12-13T02:47:22.92+00:00

@Razeen Hendricks - Did you get a chance to accept the summarized answer?

Just checking in to see if the below answer helped. If this answers your query, do click "Accept the answer” for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Answer 1

@Razeen Hendricks - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Ask: Powershell: Deserialization error on Spark pool creation

Operation returned an invalid status code 'BadRequest' Spark pool request validation failed. SparkPoolConfigContent Deserialization Failed. Content should be deserialied to SparkConfigurationInfoObject.

Solution: I've already managed to figure out the problem, which is essentially due to the incomplete json schema I used for the config when I uploaded it. It was missing the "created", "createdBy", "annotations" and "configMergeRule" fields. How any ordinary person would know this, I don't know.

See the end of https://github.com/Azure/azure-powershell/issues/20711

Once I uploaded the correctly formatted json, and then used the code to get that config. it succeeded in creating the spark pool without error.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

Thanks for your response and example @PRADEEPCHEEKATLA

I've already managed to figure out the problem, which is essentially due to the incomplete json schema I used for the config when I uploaded it. It was missing the "created", "createdBy", "annotations" and "configMergeRule" fields. How any ordinary person would know this, I don't know.

See the end of https://github.com/Azure/azure-powershell/issues/20711

Once I uploaded the correctly formatted json, and then used the code to get that config. it succeeded in creating the spark pool without error.

Please notify whoever's responsible for the documentation around this to add additional notes on the strict json format required. Your product team should add more validation rules to the upload process as well.

Share via

Powershell: Deserialization error on Spark pool creation

1 additional answer

Your answer