Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
Execution time of ADB notebook
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 8%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMG%3C/text%3E%3C/svg%3E)
Madhur Gabhane
1
Reputation point
Hello guys,
I am running an Azure databricks notebook separately, which takes 1 hour to execute fully. Whereas the same notebook when run via an ADF pipeline job (using the Databricks activity), is taking more time - 1 hour 40 mins.
Any leads on why there is a difference in execution time, when an azure databricks notebook run separately, vs via an ADF job.
Thanks in advance.
Azure Databricks
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
{count} votes
-
PRADEEPCHEEKATLA 91,576 Reputation points Moderator2021-11-16T07:33:11.757+00:00 Hello @Madhur Gabhane ,
Welcome to the Microsoft Q&A platform.
There should not be much difference running notebook via Azure Databricks and from ADF notebook activity.
When you say "Whereas the same notebook when run via an ADF pipeline job (using the Databricks activity), is taking more time - 1 hour 40 mins." are you running on the same existing interactive cluster?
In case, if you are using New job Cluster type - Could you please share the cluster configuration on both Azure Databricks and Azure Data Factory?
Azure Databricks cluster configuration:
Azure Data Factory - Azure Databricks Linked service cluster configuration:
-
Madhur Gabhane 1 Reputation point
2021-11-16T14:03:49.083+00:00 Thanks for your Comment.
I am running it on the same Interactive Cluster, still having some differences in execution time.
-
PRADEEPCHEEKATLA 91,576 Reputation points Moderator
2021-11-17T03:15:23.747+00:00 Hello @Madhur Gabhane ,
Thanks for the confirmation.
Could you please re-run and check if you still see the much difference running notebook via Azure Databricks and from ADF notebook activity?
-
Madhur Gabhane 1 Reputation point
2021-11-17T07:27:43.7+00:00 Hello Pradeep,
Yes I have rerun it for 3-4 times, and same scenario is observed.
-
PRADEEPCHEEKATLA 91,576 Reputation points Moderator
2021-11-18T09:30:39.99+00:00 Hello @Madhur Gabhane ,
This issue looks strange. For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket.
-
PRADEEPCHEEKATLA 91,576 Reputation points Moderator
2021-11-23T11:36:27.82+00:00 Hello @Madhur Gabhane ,
Did you get a chance to open a support ticket?
Sign in to comment
Sign in to answer