Share via

Monitor Databricks Notebook progress in ADF

CzarR 316 Reputation points
2022-07-14T04:35:16.527+00:00

Hi, I have a pipeline ADF that calls a databricks notebook that loads data from ADLS to Databricks tables. Now I am new Databricks and ADF. The clicked on debug and it's been running for over 2 hrs. I see there are runID's can I use those to check the progress. Problem is tables are not getting updated this whole time and the package is not failing either. I would like to see the log of what is going on. Please help. Thanks in advance.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,559 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,519 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,627 questions
0 comments
Accepted answer
  1. Pratik Somaiya 4,211 Reputation points Volunteer Moderator
    2022-07-14T06:59:49.24+00:00

    Hello @CzarR

    Yes you can monitor your Databricks notebook using ADF easily

    For that hover on the Databricks activity in the output section of your pipeline, you will see an output button, if you click on that you will find a link (runPageUrl) of the notebook which is running

    Here, you can see real time status of your notebook and can also check on which step its currently running

    Thanks!

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.