get result from databricks activity

Question

Hello,
The databricks notebook returns a table. i.e. when I execute:
...
display(df)

I now use a databricks notebook activity in data factory pipeline to execute this databricks notebook. It succeeds.
When I go to the output window, in the json format, I do not see the results returned.
1- should I be able to see the result of the execution in json?
2- How do I get the result of the databricks notebook into something like a (copy activity or another activity) to move the result to land in a .parquet file somewhere?

Thanks

Answer 1

You can directly save the dataframe data to target folder path in parquet format in ADLS. Later, simply execute this notebook in ADF.

#declare data lake path where we want to write the data
target_folder_path = 'abfss://******@adlsgen2demodatalake.dfs.core.windows.net/refined/us_covid/'

#write as parquet data
df_covid.write.format("parquet").save(target_folder_path)

Reference: https://www.mssqltips.com/sqlservertip/6499/reading-and-writing-data-in-azure-data-lake-storage-gen-2-with-azure-databricks/

---

Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav