ADF - can a value be passed from a Databricks.Python node back to an ADF pipeline and assigned to a variable

Nik Holden 26 Reputation points
2022-07-14T22:35:21.54+00:00

I have a python script that transforms and loads data to a SQL Azure table. The python script is called in an ADF pipeline. What I want to achieve is to perform an API call based on a value from the python script.

I know that by calling a Databricks Notebook I can use the dbutils.notebook.exit() function a the end of the Notebook and have the element of runOutput included in the node JSON output of the Databrick.Notebook node. But I have not been able to find a way to achieve a similar using the Python node.

The recommended solution appears to be to write the data to a file or DB table and then read it back in a lookup and pass to an API call. Is the the only way?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,136 questions
{count} votes

Accepted answer
  1. MartinJaffer-MSFT 25,996 Reputation points
    2022-07-28T21:58:28.997+00:00

    No, for the Python activity, there is no returnable output the way there is with the Notebook activity. @Nik Holden

    There is a runPageURL . This links to a page containing the outputs as in stdout and stderr. You might be able to parse something from here, but is more work compared to the Notebook activity.

    I did try the python statment sys.exit("an exit value"). This however caused the activity to return failure status. Technically this is correct, as any value other than 0 is assumed to be an error. sys.exit` is intended to raise an exception.

    On further consideration, it makes sense that there is no way to pass back values. To pass back values, there must be some hook or handler or something to receive the values. In normal garden-variety python scripts, there isn't a mechanism to pass the values out of the interpreter anyway. (Aside from printing or saving files)

    While trying to upload my python script, I accidentally turned one into a notebook by "importing" rather than saving to DBFS.


0 additional answers

Sort by: Most helpful