Attaching Spark pool during deployment in Synapse

Question

Attaching Spark pool during deployment in Synapse

Alok Thampi 151

Hello,

We have a situation where the spark pool does not get automatically attached in the target environment after deployment.
In the figure below, “testpool” is one of the spark clusters that we have in both dev and test and before deploying to test, I had a notebook, “Notebook 1” which was attached to this “testpool”.
However, after deployment, I get the below message saying that this pool is not found. The spark pool session in Test was NOT active while I did this deployment.

I would like to understand on the below.

Is this an expected scenario if the target spark pool session is closed? If so, what is the implementation approach to get the pool correctly attached to the notebook after deployment?
Is there a way that we can start the Spark pool before the deployment through scripts/DevOps plugins? I hope we don’t keep the cluster running all the time considering the cost implications

Thanks in advance,
Alok

PRADEEPCHEEKATLA 90,566 Reputation points

2022-11-08T04:26:05.74+00:00

Hello @Alok Thampi ,

Thanks for the question and using MS Q&A platform.

Could you please confirm if the Apache Spark pool named testpool exists in your workspace?

If the Apache Spark pool exists, it will be shown as attached and the status should be not started as shown below:

Hope this helps.
Alok Thampi 151 Reputation points

2022-11-08T04:40:36.067+00:00

Hello @PRADEEPCHEEKATLA ,

Thanks for the details.

Yes, the spark pool "testpool" exists in the workspace, just that it is not started.

I tried deploying back to the Test once the pool is started and then it gets attached to the notebook as expected.
The observation that I had from this activity is that if deployment happens while the pool is active in the target environment, it gets attached to the notebook without any issues. But if the pool is not active, it shows the message as above.

Since keeping the pool running all the time (especially in a Test environment) has got implications, I am looking for alternate options.

Thanks,
Alok
PRADEEPCHEEKATLA 90,566 Reputation points

2022-11-08T05:05:14.963+00:00

Hello @Alok Thampi ,

How exactly are you deploying using Azure Devops via CI/CD?
Alok Thampi 151 Reputation points

2022-11-08T05:09:51.79+00:00

Hello @PRADEEPCHEEKATLA ,

I use the Synapse deployment task and use "Validate and deploy" option for Operation Type.

Thanks,
Alok
PRADEEPCHEEKATLA 90,566 Reputation points

2022-11-08T05:58:57.2+00:00

Hello @Alok Thampi ,

This is an known issue with the are you deploying using Azure Devops.

There is a bug item open on the same issue, I will reachout to the internal team and get back to you with my findings.
Alok Thampi 151 Reputation points

2022-11-08T06:05:56.777+00:00

Thanks for this @PRADEEPCHEEKATLA , will wait for your findings.
Appreciate the quick help.

1 answer

Your answer

PRADEEPCHEEKATLA 90,566 Reputation points

2022-11-08T04:26:05.74+00:00

Hello @Alok Thampi ,

Thanks for the question and using MS Q&A platform.

Could you please confirm if the Apache Spark pool named testpool exists in your workspace?

If the Apache Spark pool exists, it will be shown as attached and the status should be not started as shown below:

Hope this helps.
Alok Thampi 151 Reputation points

2022-11-08T04:40:36.067+00:00

Hello @PRADEEPCHEEKATLA ,

Thanks for the details.

Yes, the spark pool "testpool" exists in the workspace, just that it is not started.

I tried deploying back to the Test once the pool is started and then it gets attached to the notebook as expected.
The observation that I had from this activity is that if deployment happens while the pool is active in the target environment, it gets attached to the notebook without any issues. But if the pool is not active, it shows the message as above.

Since keeping the pool running all the time (especially in a Test environment) has got implications, I am looking for alternate options.

Thanks,
Alok
PRADEEPCHEEKATLA 90,566 Reputation points

2022-11-08T05:05:14.963+00:00

Hello @Alok Thampi ,

How exactly are you deploying using Azure Devops via CI/CD?
Alok Thampi 151 Reputation points

2022-11-08T05:09:51.79+00:00

Hello @PRADEEPCHEEKATLA ,

I use the Synapse deployment task and use "Validate and deploy" option for Operation Type.

Thanks,
Alok
PRADEEPCHEEKATLA 90,566 Reputation points

2022-11-08T05:58:57.2+00:00

Hello @Alok Thampi ,

This is an known issue with the are you deploying using Azure Devops.

There is a bug item open on the same issue, I will reachout to the internal team and get back to you with my findings.
Alok Thampi 151 Reputation points

2022-11-08T06:05:56.777+00:00

Thanks for this @PRADEEPCHEEKATLA , will wait for your findings.
Appreciate the quick help.

Answer 1

Jan Kučera 0

Hello @Alok Thampi , was this resolved? I similar issue.

Share via

Attaching Spark pool during deployment in Synapse

1 answer

Your answer