Share via

ADF Dataflow Debug is not refreshing data

K J, Sujanya 0 Reputation points
2024-07-03T10:52:32.68+00:00

Hi All,

I am trying to run my Dataflow pipeline which consists of an inline dataset pointing to Common Data Model and When I run my pipeline for the first time data is loaded properly to sink but it is not fetching latest data in the consecutive runs in debug mode.
How to fix this issue?

As per my research there is such existing issue and there were suggestions to change the source name but it is not working for me.

Will we have the same issue if we publish and trigger the pipeline from live environment.?

Thank you in advance.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,158 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Deleted

    This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

    Comments have been turned off. Learn more

  2. Sujanya KJ 0 Reputation points
    2024-07-03T19:32:38.1+00:00

    Data was dumped into adls gen 2 using azure synapse connector by dataverse . Later we built the pipeline with upsert logic using dataflow to ingeat thw data into datawarehouse from adls.

    The issue which i am facing seems intermittent, sometimw data is being uploaded without any issue in the debug mode using upsert logic written. Sometimes it is not as thw data preview at the source end dataflow itself ia showing the older data. I am suspecting it could be cache related issue.

    Is there any resolution for this.

    Or any idea will we face the same iasue at trigger mode as well.

    0 comments