I can't delete a Synapse Spark Pool

Question

When I try to delete a Synapse Spark Pool, I get this error:

$ az synapse spark pool delete --workspace-name myws --resource-group myrg -n poolname
Are you sure you want to perform this operation? (y/n): y
(DeleteSparkComputeAdfError) The document cannot be deleted since it is referenced by Read and write data from Azure Data Lake Storage Gen2.
Code: DeleteSparkComputeAdfError
Message: The document cannot be deleted since it is referenced by Read and write data from Azure Data Lake Storage Gen2.

How would a data lake reference a Spark Pool?

I have removed all reference to this pool from any notebook, so that shouldn't be it.

Answer 1

@Mathieu Longtin - Thanks for the question and using MS Q&A platform.

It seems like the Synapse Spark Pool you are trying to delete is being referenced by "Read and write data from Azure Data Lake Storage Gen2". This means that there are some dependencies on the pool that need to be removed before you can delete it.

To resolve the issue, we suggest removing these references identified and try deletion again. The spark pool’s reference in notebooks should be removed from all branches. You may need to remove the spark pool’s reference in each branch, publish changes, and delete the spark pool again.

To remove the dependencies, you can try the following steps:

1. Go to the Azure Synapse Analytics workspace in the Azure portal.
2. Click on the "Data" tab and then select "Linked services".
3. Find the linked service that is using the Synapse Spark Pool you want to delete and click on it.
4. In the linked service details page, remove the reference to the Synapse Spark Pool you want to delete.
5. Save the changes and try deleting the Synapse Spark Pool again.

If you are still unable to delete the Synapse Spark Pool after removing the dependencies, please use below scripts:

To list the pool’s references in notebooks in workspace, run the following command:

az synapse notebook list --workspace-name <workspaceName> | jq '.[] | "Notebook: " + .name + ", Pool: " + .properties.bigDataPool.referenceName' | grep <sparkPoolName>

To list the pool’s references in Spark jobs definitions, run the following command:

az synapse spark-job-definition list --workspace-name <workspaceName> | jq '.[] | "Spark Job: " + .name + ", Pool: " + .properties.targetBigDataPool.referenceName' | grep <sparkPoolName>

For more details, refer to MS Q&A thread addressing similar issue: https://learn.microsoft.com/en-us/answers/questions/1848656/azure-apache-spark-pool-stuck-in-resourceinunupdat

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.