Databricks Python wheel tasks submission - shared job cluster feature us not suppported in runs/submit API error

Question

I am creating a job with a handful of tasks that have some dependency on each other. Using Python whl submission for all the tasks and whenever I try to reuse a cluster, which I am doing by defining "job_clusters" fields with "new_cluster" and "job_cluster_key" while also adding a "job_cluster_key" field in the tasks that is the same value as the "job_cluster_key" in the new cluster. I get this error:

{
"error_code": "INVALID_PARAMETER_VALUE",
"message": "Shared job cluster feature is not supported in runs/submit API."
}

It works when I do not specify a "job_clusters" and create a new cluster each time for every task in the request. Using 2.1 api /api/2.1/jobs/runs/submit to create a job. I want to do cluster reuse for my tasks within this job request

Answer 1

Hello @Fayez Ahmed (US) ,

Thanks for the question and using MS Q&A platform.

This is an excepted behaviour.

To submit a one-time run of a multi-task format job, use the tasks field in JobSettings to specify settings for each task, including clusters. Clusters must be set at the task level when submitting a multi-task format job because the runs submit request does not support shared job clusters.

For more details, refer to Jobs API updates - Runs submit.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators