How to fix Synapse data flow not updating data in delta lake?

JaneTeo-1997 0 Reputation points
2023-03-17T09:05:27.8466667+00:00

Hi,

A dataflow mapping in Synapse writing to a delta sink should update rows based on an Alter row transformation. In the data preview of the debug session I can see that the row should be updated, however after running the pipeline the record is not being updated in the Delta lake.

This is the data preview of the sink in data flow.

User's image

Does anyone know how to solve this?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,042 questions
{count} votes

2 answers

Sort by: Most helpful
  1. JaneTeo-1997 0 Reputation points
    2023-03-21T06:44:51.7833333+00:00

    @MartinJaffer-MSFT Thanks for the tip to check the diagnostics. I had checked it before, I didn't check the source step. Turned out I still had sampling enabled.

    Thank you both @Subashri Vasudevan


  2. MartinJaffer-MSFT 26,106 Reputation points
    2023-03-21T16:23:16.4866667+00:00

    @JaneTeo-1997 What if you enable upsert?

    Also, you can find more diagnostic information if you go to monitoring tab, find the pipeline run, go into it and get the dataflow run details like rows affected.

    User's image

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.