Azure data factory - Azure Databricks Notebook

Sharma, Abhinav 21 Reputation points
2021-08-10T12:17:57.967+00:00

I need to load output of "Look Up" activity into "Azure Databricks Notebook". Is it possible to load output of ADF Look Up activity directly into Azure databricks notebook ?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,693 questions
0 comments No comments
{count} votes

Accepted answer
  1. KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee
    2021-08-18T16:32:28.33+00:00

    Hi @Anonymous ,

    As per my research and discussion with internal teams, here are few workarounds you could try to overcome the limitations discussed in our previous comments:

    1. Instead of passing the actual data from the Lookup into the notebook you could try passing the query used in the Lookup to the notebook and have the notebook read from the SQL DB.
    2. Otherwise, you could stage that data in a parquet file and pass the location string into the notebook as well.
    3. The third option is to use Mapping data flows instead of Databricks if it fits your transformation logic.

    Hope this info helps.

    ----------

    • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification.
    1 person found this answer helpful.

1 additional answer

Sort by: Most helpful
  1. Vaibhav Chaudhari 38,636 Reputation points
    2021-08-10T14:05:04.697+00:00

    It is possible to pass the value to a parameter which can be used inside Databricks notebook

    Reference - https://learn.microsoft.com/en-us/azure/data-factory/transform-data-using-databricks-notebook#create-a-pipeline

    ----------

    Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav

    1 person found this answer helpful.