Returning result from azure databricks to azure datafacotry

Question

I am new to using Azure Databricks and ADF.

I have a table in azure databricks. I would like to run a select query on this table and return the result of my select query to ADF. When I searched on how this can be done, I see we can use dbutils.notebook.exit() for returning the output. Will it work if I write

dbutils.notebook.exit(spark.sql('select * from Invoice').toJSON().collect())

Is there a limitation on how much data can be returned back?

If this is not the best way to do it, can someone please provide some suggestions on how I can get the result of the query as I need to pass this dataset to an azure function for further processing?

Answer 1

Hello @Karpagam Gurumurthy ,
Thanks for the ask and using Microsoft Q&A platform .
As we understand the ask here is how to result from azure databricks to azure datafactory . Please do let me know if that not accurate
The better approach will be write the table content to a cloud storage and then read the same from ADF .
Steps :

1. Call a notebook from ADF , which reads the table & writes to a blob on clod storage
2. Add a lookup activity in ADF and read the same blob .

Please do let me if you have any queries .
Thanks
Himanshu

-------------------------------------------------------------------------------------------------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators