How to pass parameters from Data Factory to Delta Live Table pipeline?

Question

How to pass parameters from Data Factory to Delta Live Table pipeline?

Manjula Ganesapandian 0

Can i pass variables from data factory to change the configuration of the delta live table pipeline?

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-08-23T23:54:22.4+00:00
@Manjula Ganesapandian Thanks for using Microsoft Q&A forum and posting your query.

Could you please provide more information on how you are trying/connecting to Databricks Delta Live Tables Pipeline to update the configuration? Are you using REST API using Web activity or some other way?

If you are using Web activity, then have you got a chance to try passing variables/parameters from POSTMan before trying it in ADF directly? If not, I would recommend trying it out in PostMan and then once successful, we can implement the same in ADF Web activity.

Irrespective, if your ask is how to pass ADF variables value to Web activity POST/PUT Body request, then below is sample how you have to configure your POST Body :

{ "DataFactoryName": "@{pipeline().DataFactory}", "PipelineName": "@{pipeline().Pipeline}", "Subject": "@{variables('EmailSubject')}", "ErrorMessage": "Everything is okey-dokey!", "EmailTo": "******@myEmail.com" }

You can refer to this SO thread for more info: https://stackoverflow.com/questions/60505409/adf-v2-web-post-method-using-dynamic-content-and-variable

Please let us know how it goes.

Thank you
Manjula Ganesapandian 0 Reputation points

2023-08-24T13:26:50.2833333+00:00
Thank you for the response, Kranthi.

Let me explain better. I have the notebook with the DLT code and I created a DLT pipeline to execute it. The execution is successful. In the DLT UI, I added the below:

In the notebook, i added the below code and was able to use the variable for processing.

sourceName = spark.conf.get("ingestionSourceName")

Now, through ADF, I'm trying to override the value of ingestionSourceName.

In ADF, I have a web activity to connect to the API with POST. And I'm passing the below in the body.

{"full_refresh":"true","configuration":{"ingestionSourceName":"RPM"}}

But it does not override, though i see the full_refresh portion of it working. So my question is, what am i doing wrong and how can i do the override? Thank you for your help with this.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-09-01T05:43:30.7833333+00:00

@Manjula Ganesapandian Thank you for the additional details. I'm testing this scenario as I haven't tried before and will keep you posted as soon as I have my findings ready.

1 answer

Your answer

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-08-23T23:54:22.4+00:00

@Manjula Ganesapandian Thanks for using Microsoft Q&A forum and posting your query.

Could you please provide more information on how you are trying/connecting to Databricks Delta Live Tables Pipeline to update the configuration? Are you using REST API using Web activity or some other way?

If you are using Web activity, then have you got a chance to try passing variables/parameters from POSTMan before trying it in ADF directly? If not, I would recommend trying it out in PostMan and then once successful, we can implement the same in ADF Web activity.

Irrespective, if your ask is how to pass ADF variables value to Web activity POST/PUT Body request, then below is sample how you have to configure your POST Body :

{ "DataFactoryName": "@{pipeline().DataFactory}", "PipelineName": "@{pipeline().Pipeline}", "Subject": "@{variables('EmailSubject')}", "ErrorMessage": "Everything is okey-dokey!", "EmailTo": "******@myEmail.com" }

You can refer to this SO thread for more info: https://stackoverflow.com/questions/60505409/adf-v2-web-post-method-using-dynamic-content-and-variable

Please let us know how it goes.

Thank you
Manjula Ganesapandian 0 Reputation points

2023-08-24T13:26:50.2833333+00:00

Thank you for the response, Kranthi.

Let me explain better. I have the notebook with the DLT code and I created a DLT pipeline to execute it. The execution is successful. In the DLT UI, I added the below:

In the notebook, i added the below code and was able to use the variable for processing.

sourceName = spark.conf.get("ingestionSourceName")

Now, through ADF, I'm trying to override the value of ingestionSourceName.

In ADF, I have a web activity to connect to the API with POST. And I'm passing the below in the body.

{"full_refresh":"true","configuration":{"ingestionSourceName":"RPM"}}

But it does not override, though i see the full_refresh portion of it working. So my question is, what am i doing wrong and how can i do the override? Thank you for your help with this.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-09-01T05:43:30.7833333+00:00

@Manjula Ganesapandian Thank you for the additional details. I'm testing this scenario as I haven't tried before and will keep you posted as soon as I have my findings ready.

Answer 1

@Manjula Ganesapandian I have a workaround to this. Instead of calling the post to run the delta live pipeline, call the edit pipeline rest API - https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/api-guide#--edit-a-pipeline and update the pipeline with your new config. This will be a synchronous update. Then call the Start pipeline api https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/api-guide#--start-a-pipeline-update. This should fine for your scenario as only 1 instance of the pipeline can be running at a time.

Share via

How to pass parameters from Data Factory to Delta Live Table pipeline?

1 answer

Your answer