Azure data factory - Azure Databricks Notebook

Question

I need to load output of "Look Up" activity into "Azure Databricks Notebook". Is it possible to load output of ADF Look Up activity directly into Azure databricks notebook ?

Answer 1

Hi @Anonymous ,

As per my research and discussion with internal teams, here are few workarounds you could try to overcome the limitations discussed in our previous comments:

1. Instead of passing the actual data from the Lookup into the notebook you could try passing the query used in the Lookup to the notebook and have the notebook read from the SQL DB.
2. Otherwise, you could stage that data in a parquet file and pass the location string into the notebook as well.
3. The third option is to use Mapping data flows instead of Databricks if it fits your transformation logic.

Hope this info helps.

----------

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
• Want a reminder to come back and check responses? Here is how to subscribe to a notification.

Answer 2

It is possible to pass the value to a parameter which can be used inside Databricks notebook

Reference - https://learn.microsoft.com/en-us/azure/data-factory/transform-data-using-databricks-notebook#create-a-pipeline

----------

Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav