Hi Expert, I am running in select count(* ) from employee on databricks notebook getting output as 1000 wanted to pass that output to another lookup.. count(* ) from employee output =10000 how can i do it

Hi Shambhu Rai , Thankyou for using Microsoft Q&A platform and thanks for posting your question here. As per my understanding, you are trying to pass the output of databricks notebook into lookup activity in ADF pipeline. Please let me know if that is not the correct understanding. In your notebook, you may call dbutils.notebook.exit("returnValue") and corresponding "returnValue" will be returned to the service. You can consume the output in the service by using expression such as @{activity('databricks notebook activity name').output.runOutput} If you are passing JSON object you can retrieve values by appending property names. Example: @{activity('databricks notebook activity name').output.runOutput.PropertyName} For more details, kindly check the below documentation: Passing parameters between notebooks and pipelines Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou

notebook output in lookup activity adf

Accepted answer

AnnuKumari-MSFT 32,736 Reputation points Microsoft Employee

2023-09-26T09:38:03.9933333+00:00
Hi Shambhu Rai ,

Thankyou for using Microsoft Q&A platform and thanks for posting your question here.

As per my understanding, you are trying to pass the output of databricks notebook into lookup activity in ADF pipeline. Please let me know if that is not the correct understanding.

In your notebook, you may call dbutils.notebook.exit("returnValue") and corresponding "returnValue" will be returned to the service.

You can consume the output in the service by using expression such as @{activity('databricks notebook activity name').output.runOutput}

If you are passing JSON object you can retrieve values by appending property names. Example: @{activity('databricks notebook activity name').output.runOutput.PropertyName}

For more details, kindly check the below documentation: Passing parameters between notebooks and pipelines

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou
Please sign in to rate this answer.

2 people found this answer helpful.
Shambhu Rai 1,411 Reputation points

2023-09-26T15:32:19.4633333+00:00

Hi Expert,

the question is how to pass this parameter in lookup transformatoin...pLese check the screenshot before answer

Shambhu Rai 1,411 Reputation points

2023-09-27T04:09:55.2833333+00:00

suggestion please

AnnuKumari-MSFT 32,736 Reputation points Microsoft Employee

2023-09-27T05:11:11.4533333+00:00

Hi Shambhu Rai ,

Yes I saw the screenshot. Could you please tell which datastore the lookup dataset is pointing to ? In case its pointing to a sql database, you can use the above mentioned expression @{activity('databricks notebook activity name').output.runOutput.PropertyName} within the sql query by selecting the query option or else you can use set variable activity to consume the notebook activity output .

Shambhu Rai 1,411 Reputation points

2023-09-27T05:52:04.6433333+00:00

the select statement is expected from notebook as the previous lookup is only passing parameters the same like notebook then how it will do a lookup??

example: select emp from emp,@{activity('Previouslookupparametername').output

how we can get lookup parameter as select statement or any dummy variable in select to get output

Shambhu Rai 1,411 Reputation points

2023-09-27T05:58:26.6333333+00:00

it connecting to hivemetastore from databricks...is there any way we can connect with it from ADF..it will solve the problem

AnnuKumari-MSFT 32,736 Reputation points Microsoft Employee

2023-09-27T06:49:51.3333333+00:00

Hi Shambhu Rai ,

Kindly write the expression by opening 'Add dynamic content' so that it will treat the expression as a dynamic statement instead of static string and see it it works .

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou

Shambhu Rai 1,411 Reputation points

2023-09-27T06:53:28.25+00:00

thanks for solution
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

notebook output in lookup activity adf

0 additional answers

Your answer