Powershell: Deserialization error on Spark pool creation

Razeen Hendricks 20 Reputation points
2023-12-04T12:25:49.9933333+00:00

Hi,

I'm trying to create a spark pool via Powershell but get the following error:

Operation returned an invalid status code 'BadRequest' Spark pool request validation failed. SparkPoolConfigContent Deserialization Failed. Content should be deserialied to SparkConfigurationInfoObject.

The following is the script I'm using:

$config = Get-AzSynapseSparkConfiguration -WorkspaceName $WORKSPACE -Name $CONFIG_NAME  

New-AzSynapseSparkPool -ResourceGroupName $RESOURCE_GROUP_NAME -WorkspaceName $WORKSPACE -Name $SPARK_POOL_NAME `
     -EnableDynamicExecutorAllocation -MinExecutorCount $MIN_EXECUTOR_COUNT -MaxExecutorCount $MAX_EXECUTOR_COUNT `
     -SparkVersion $SPARK_VERSION -NodeSize $NODE_SIZE -AutoScaleMinNodeCount $MIN_NODE_COUNT -AutoScaleMaxNodeCount $MAX_NODE_COUNT `
     -EnableAutoPause -AutoPauseDelayInMinute $PAUSE_DELAY_MINS -SparkConfiguration $config 

The documentation I've followed is: https://learn.microsoft.com/en-us/powershell/module/az.synapse/new-azsynapsesparkpool?view=azps-10.4.1#example-6

Why is the error being encountered? How could I get this to work?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,673 questions
{count} votes

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 85,351 Reputation points Microsoft Employee
    2023-12-07T08:53:03.22+00:00

    @Razeen Hendricks - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

    Ask: Powershell: Deserialization error on Spark pool creation

    Operation returned an invalid status code 'BadRequest' Spark pool request validation failed. SparkPoolConfigContent Deserialization Failed. Content should be deserialied to SparkConfigurationInfoObject.

    Solution: I've already managed to figure out the problem, which is essentially due to the incomplete json schema I used for the config when I uploaded it. It was missing the "created", "createdBy", "annotations" and "configMergeRule" fields. How any ordinary person would know this, I don't know.

    See the end of https://github.com/Azure/azure-powershell/issues/20711

    Once I uploaded the correctly formatted json, and then used the code to get that config. it succeeded in creating the spark pool without error.

    If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

    If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.


    Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

    0 comments No comments

1 additional answer

Sort by: Most helpful
  1. Razeen Hendricks 20 Reputation points
    2023-12-07T07:49:57.99+00:00

    Thanks for your response and example @PRADEEPCHEEKATLA-MSFT

    I've already managed to figure out the problem, which is essentially due to the incomplete json schema I used for the config when I uploaded it. It was missing the "created", "createdBy", "annotations" and "configMergeRule" fields. How any ordinary person would know this, I don't know.

    See the end of https://github.com/Azure/azure-powershell/issues/20711

    Once I uploaded the correctly formatted json, and then used the code to get that config. it succeeded in creating the spark pool without error.

    Please notify whoever's responsible for the documentation around this to add additional notes on the strict json format required. Your product team should add more validation rules to the upload process as well.

    0 comments No comments