ADF - can a value be passed from a Databricks.Python node back to an ADF pipeline and assigned to a variable

Question

ADF - can a value be passed from a Databricks.Python node back to an ADF pipeline and assigned to a variable

Nik Holden 31

I have a python script that transforms and loads data to a SQL Azure table. The python script is called in an ADF pipeline. What I want to achieve is to perform an API call based on a value from the python script.

I know that by calling a Databricks Notebook I can use the dbutils.notebook.exit() function a the end of the Notebook and have the element of runOutput included in the node JSON output of the Databrick.Notebook node. But I have not been able to find a way to achieve a similar using the Python node.

The recommended solution appears to be to write the data to a file or DB table and then read it back in a lookup and pass to an API call. Is the the only way?

MartinJaffer-MSFT 26,161 Reputation points

2022-07-18T20:24:58.033+00:00

Hello @Nik Holden and welcome to Microsoft Q&A.

I understand you want to pass some data back from Databricks Notebook to ADF. I see you have done research and found the 2-3 methods I would have recommended.

However I am I little bit confused when you talk about "Databricks.Python node". Is this something different than the usual Python-language Notebook running on Databricks cluster?

Ah, are you referring to single-node cluster? This appears new and I haven't touched it yet.

I do vaugely recall once trying to use a Web activity to peek at the driver logs. Or something like that. I mention that because single-node cluster properties says:

All stderr, stdout, and log4j log output is saved in the driver log.
MartinJaffer-MSFT 26,161 Reputation points

2022-07-18T20:25:59.973+00:00

I tested a notebook on single node cluster, and I got the expected runOutput
Nik Holden 31 Reputation points

2022-07-18T22:53:36.677+00:00

Thanks for replying.

I may be referring to the objects that make up an ADF pipeline incorrectly. I call them actions (e.g a Lookup action) but I see them referred to as nodes. This is not a not a node in a Databricks cluster.

I am trying to achieve this without using a Notebook. All of our code is in py files so we use the Python node in the 3 options (Notebook, Jar, Python) under the Databricks collection of nodes. I have not been able to return a custom element in the python node output to use as an input for other nodes.
MartinJaffer-MSFT 26,161 Reputation points

2022-07-19T17:35:16.627+00:00

Thank you for the clarification @Nik Holden .

I haven't tried with that before. I'll see what I can find.

Answer accepted by question author

0 additional answers

Your answer

MartinJaffer-MSFT 26,161 Reputation points

2022-07-18T20:24:58.033+00:00

Hello @Nik Holden and welcome to Microsoft Q&A.

I understand you want to pass some data back from Databricks Notebook to ADF. I see you have done research and found the 2-3 methods I would have recommended.

However I am I little bit confused when you talk about "Databricks.Python node". Is this something different than the usual Python-language Notebook running on Databricks cluster?

Ah, are you referring to single-node cluster? This appears new and I haven't touched it yet.

I do vaugely recall once trying to use a Web activity to peek at the driver logs. Or something like that. I mention that because single-node cluster properties says:

All stderr, stdout, and log4j log output is saved in the driver log.
MartinJaffer-MSFT 26,161 Reputation points

2022-07-18T20:25:59.973+00:00

I tested a notebook on single node cluster, and I got the expected runOutput
Nik Holden 31 Reputation points

2022-07-18T22:53:36.677+00:00

Thanks for replying.

I may be referring to the objects that make up an ADF pipeline incorrectly. I call them actions (e.g a Lookup action) but I see them referred to as nodes. This is not a not a node in a Databricks cluster.

I am trying to achieve this without using a Notebook. All of our code is in py files so we use the Python node in the 3 options (Notebook, Jar, Python) under the Databricks collection of nodes. I have not been able to return a custom element in the python node output to use as an input for other nodes.
MartinJaffer-MSFT 26,161 Reputation points

2022-07-19T17:35:16.627+00:00

Thank you for the clarification @Nik Holden .

I haven't tried with that before. I'll see what I can find.

Answer 1

No, for the Python activity, there is no returnable output the way there is with the Notebook activity. @Nik Holden

There is a runPageURL . This links to a page containing the outputs as in stdout and stderr. You might be able to parse something from here, but is more work compared to the Notebook activity.

I did try the python statment sys.exit("an exit value"). This however caused the activity to return failure status. Technically this is correct, as any value other than 0 is assumed to be an error. sys.exit` is intended to raise an exception.

On further consideration, it makes sense that there is no way to pass back values. To pass back values, there must be some hook or handler or something to receive the values. In normal garden-variety python scripts, there isn't a mechanism to pass the values out of the interpreter anyway. (Aside from printing or saving files)

While trying to upload my python script, I accidentally turned one into a notebook by "importing" rather than saving to DBFS.

Nik Holden 31 Reputation points

2022-07-28T22:08:14.357+00:00

Thank you for your reply and effort spent looking at this. This is what I thought.

Share via

ADF - can a value be passed from a Databricks.Python node back to an ADF pipeline and assigned to a variable

0 additional answers

Your answer