Triggering a Databricks Delta Live Table from Azure Data Factory resets the whole tables. How do I disable that?

Question

Triggering a Databricks Delta Live Table from Azure Data Factory resets the whole tables. How do I disable that?

Anonymous

I have created a pipeline in Azure Data Factory that triggers a Delta Live Table in Azure Databricks through a Web activity mentioned here in the Microsoft documentation. My problem is that when I trigger my DLT from ADF, it resets the whole tables, meaning that my data becomes unavailable during the pipeline execution. Here's how it looks like with this extra step: User's image

However, when I run it directly from the Databricks UI, the tables will not get reset and the data is available during the execution of my pipeline. Here's how it looks like: User's image

I would like to have the same behavior in ADF, as I have when trigger the pipeline directly from the Databricks UI. I don't want to have this additional "resetting tables" step in my DLT pipeline when I trigger it from ADF. Additional information: My Delta Live Table is a streaming pipeline, meaning that it will only pick up the new files that's been added to my source since the last time it ran. That's exactly my problem; resetting the whole tables pick every single historical file, as if I am running the pipeline for the first time.

Vidhya Sagar Karthikeyan 396 Reputation points

2023-04-05T09:37:55.33+00:00

I second comment. If you notice ADF doesn't have custom task for this, it uses the web activity and it just invokes the databricks API to kick start a pipeline. When you use full refresh in the body then databricks will reset the tables and populate it. Since you don't want to do a full refresh just pass {} empty braces in the body and that should just pick from where it's left last time. Mark my comment as answer if it helps you.
PRADEEPCHEEKATLA 90,656 Reputation points Moderator

2023-04-10T08:04:14.8966667+00:00

Moein Torabi - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
PRADEEPCHEEKATLA 90,656 Reputation points Moderator

2023-04-19T08:01:18.5266667+00:00

Moein Torabi - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Accepted answer

0 additional answers

Your answer

Vidhya Sagar Karthikeyan 396 Reputation points

2023-04-05T09:37:55.33+00:00

I second comment. If you notice ADF doesn't have custom task for this, it uses the web activity and it just invokes the databricks API to kick start a pipeline. When you use full refresh in the body then databricks will reset the tables and populate it. Since you don't want to do a full refresh just pass {} empty braces in the body and that should just pick from where it's left last time. Mark my comment as answer if it helps you.
PRADEEPCHEEKATLA 90,656 Reputation points Moderator

2023-04-10T08:04:14.8966667+00:00

Moein Torabi - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
PRADEEPCHEEKATLA 90,656 Reputation points Moderator

2023-04-19T08:01:18.5266667+00:00

Moein Torabi - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

PRADEEPCHEEKATLA 90,656 Moderator

Moein Torabi - Thanks for the question and using MS Q&A platform.

How is your Web Activity request constructed? Do you use "{"full_refresh": "true"}." or {} in your request body? If true, the Delta Live Tables system resets all tables that are resettable before running the pipeline. - Delta Live Tables API guide - Azure Databricks | Microsoft Learn

PRADEEPCHEEKATLA 90,656 Reputation points Moderator

2023-04-12T07:06:29.6166667+00:00

Moein Torabi - Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

Triggering a Databricks Delta Live Table from Azure Data Factory resets the whole tables. How do I disable that?

0 additional answers

Your answer