databricks job is getting skipped

Shambhu Rai 1,411 Reputation points
2024-01-10T00:04:41.6466667+00:00

currently i am running 3 different job using queue disabled and concurrent run is 1 but sometimes job is getting skipped and if i enable the queue it is waiting for another job to finish

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,657 questions
{count} votes

1 answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA 90,661 Reputation points Moderator
    2024-01-10T02:03:27.8033333+00:00

    @Shambhu Rai - Thanks for the question and using MS Q&A platform.

    It seems like you are experiencing issues with Azure Databricks job skipping. There could be several reasons why this is happening. One possible reason is that the job is failing to start due to an error in the job configuration or the cluster configuration. Another possible reason is that the job is being preempted by another job that has a higher priority.

    To troubleshoot this issue, you can start by checking the job logs to see if there are any error messages or warnings that might indicate why the job is being skipped. You can also check the cluster logs to see if there are any issues with the cluster that might be causing the job to fail.

    If you are using a job queue, you can try increasing the number of concurrent runs to see if that helps. You can also try adjusting the priority of the job to see if that helps.

    For more details, refer to Troubleshoot and repair job failures.

    Hope this helps. Do let us know if you any further queries.


    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.