Synapse Spark pools get omitted in Synapse Workspace template

Minh NGUYEN 126

Hi,

I'm using Azure DevOps to deploy Synapse Workspace in a release pipeline using 'Synapse workspace deployment' plugin. There's a Spark pool in my source Workspace but the target Workspace deployment fails half way through, when deploying a Spark job definition, with error: "The document creation or update failed because of invalid reference \'sparkpool1\'. Earlier in the logs, it deployed Dedicated SQL pool and SQL scripts successfully.

So I looked at the TemplateForWorkspace.json on the workspace_publish branch on GIT repo. There's no declaration of SparkPool1 as a resource. It is only referred once when it comes to the Spark job definition as "referenceName" in "properties".

It seems like an issue when generating the template when publishing the source Workspace.

Could you please advise on how I can solve this?

MartinJaffer-MSFT 26,021 Reputation points

2021-01-21T07:27:31.757+00:00

Hello @Minh NGUYEN and welcome to Microsoft Q&A. Thank you for bringing this to our attention.

If I understand correctly, the deployment first creates the spark pool successfully, but then later fails when creating the spark job definitions.

What name is the spark pool created with? Sounds like it does not match the one in the spark job definition.

I will reach out internally for more information on the issue.
Minh NGUYEN 126 Reputation points

2021-01-21T22:25:11.343+00:00

Hi @MartinJaffer-MSFT ,

I've got 1 dedicated SQL pool, and 1 Spark pool named 'sparkpool1'. The dedicated SQL Pool was created successfully. It can deploy the SQL pool and those associated SQL scripts. Once it got to the Spark job, it failed with the mentioned error about the Spark pool named 'sparkpool1'.

The ARM template that was generated in the 'workspace_publish' branch, does declare the SQL pool, but not the Spark pool. So I guess that for some reason it ignored the Spark pool earlier in the script. Then when it came to deploy the Spark job, it doesn't know what 'sparkpool1' is.
Minh NGUYEN 126 Reputation points

2021-01-28T11:30:26.837+00:00

Hi @MartinJaffer-MSFT ,

Have you got any updates for me on this issue?
Minh NGUYEN 126 Reputation points

2021-02-01T17:54:26.1+00:00

Hi @MartinJaffer-MSFT , @PRADEEPCHEEKATLA-MSFT , @KranthiPakala-MSFT and Microsoft supports,

Have you got any updates for me on this issue?
Just to briefly reiterate, it seems that after studio publishing, the TemplateForWorkspace.json does not include the Spark Pools created in Synapse Workspace.
If it is a bug then it might be critical. If not, could you please let me know what I'm missing here?
MartinJaffer-MSFT 26,021 Reputation points

2021-02-04T01:06:42.887+00:00

Thank you for the reminder. I am following up hard now to get a resolution.

1 answer

MartinJaffer-MSFT 26,021 Reputation points

2021-03-03T18:44:07.577+00:00

@Minh NGUYEN I heard back.

The workspace deployment task in the devops is only for the artifacts deployment(non-arm call). For pools, the workspace resources(arm call), you need to have them ready before the artifacts deployment.
Users can custom the parameters in the parameter file, and parameterize the pools’ properties.
Please sign in to rate this answer.

1 person found this answer helpful.
Minh NGUYEN 126 Reputation points

2021-04-09T10:42:20.64+00:00

Hi @MartinJaffer-MSFT ,

Thanks for your answer. So according to your explanation, the pools (ARM calls) need to be ready before artifact deployment
However I notice that the dedicated (provisioned) SQL pools (also ARM call if I'm not mistaken) get deployed successfully in the task. This is inconsistent compare to the Spark pools. Why is that?
Sign in to comment