ADF Dataflow - Bronze Delta Table to Silver Delta Table Merges delete completely bronze table

Edgar Bahilo Rodríguez 6 Reputation points
2023-12-13T19:53:09.01+00:00

Hi,

I have the following pipelines:

  1. Onprem SQL Server Source to Landing Zone (in parquet):
    User's image
  2. Landing to Bronze table (delta) where I am specifying to delete the parquet source files: User's image

User's image

  1. Bronze to Silver where I want to perform upserts:
    User's image

For some reason the second dataflow (where I am not specifiying to delete any source files, in fact you do not have this option in the UI) deletes all the files from the bronze delta table.

Is there any reason for this default behaviour???

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,462 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,429 questions
{count} vote

1 answer

Sort by: Most helpful
  1. Edgar Bahilo Rodríguez 6 Reputation points
    2023-12-15T17:22:28.4466667+00:00

    Hi BhargavaGunnam-MSFT any chance to look at this?


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.