Tumbling window trigger in copy pipeline seems to be copying more data than present in the window.

Question

Tumbling window trigger in copy pipeline seems to be copying more data than present in the window.

Swetha Seelam 0 Microsoft Employee

I used the 'Ingest' option in Azure data factory to enable continuous ingestion from one table to another table using Tumbling window trigger. I set the interval in the Tumbling window trigger to 15 mins and the window start and end time for each trigger are as expected.
User's image

But the pipeline tries to copy a lot more data than what is present in this 15 min interval and it doesn't complete, it just keeps trying to copy.
Any thoughts on how to fix this?

Smaran Thoomu 34,960 Reputation points Microsoft External Staff Moderator

2024-03-26T08:12:49.3666667+00:00
Hi @Swetha Seelam

Thank you for reaching out to the community forum. I understand that you are facing an issue with the Tumbling window trigger in Azure Data Factory, where the pipeline is copying more data than what is present in the 15-minute interval, and it doesn't complete, it just keeps trying to copy.

To fix this issue, you can try the following steps:

Check if the source data is being updated frequently. If the source data is being updated frequently, it can cause the pipeline to copy more data than expected. In this case, you can try using the watermark column to track the last ingestion time and only copy the data that has been updated since the last ingestion.

Check if the pipeline is configured correctly. Ensure that the pipeline is configured to use the Tumbling window trigger with the correct interval and window start and end time.

Check if the pipeline is using the correct data slice. If the pipeline is using the wrong data slice, it can cause the pipeline to copy more data than expected. In this case, you can try using the "Data preview" option in the pipeline to verify the data slice.

Check if the pipeline is using the correct data format. If the pipeline is using the wrong data format, it can cause the pipeline to copy more data than expected. In this case, you can try using the "Data preview" option in the pipeline to verify the data format.

If the issue persists, please provide more details about your pipeline configuration, source data, and destination data. This will help us to provide a more accurate solution.

I hope this helps. Let me know if you have any further questions.
Swetha Seelam 0 Reputation points Microsoft Employee

2024-03-26T18:49:06.0733333+00:00

Hi @Smaran Thoomu , thank you for providing some ideas on this. The source data isn't being updated frequently and the pipeline is configured to use the Tumbling window trigger.

What do you mean by correct data slice and correct data format? Could you please elaborate on how to approach this?
Smaran Thoomu 34,960 Reputation points Microsoft External Staff Moderator

2024-03-28T09:48:09.3366667+00:00

@Swetha Seelam By "correct data slice," I mean that the data being ingested by the pipeline should be limited to the specific time window defined by the Tumbling window trigger. This ensures that only the data that falls within the specified time window is ingested.

By "correct data format," I mean that the data being ingested should be in the correct format expected by the pipeline. For example, if the pipeline is expecting data in a specific file format or data type, the source data should be in that format or data type.

After discussing this matter offline, we're contacting our internal team to gather more details regarding your question. We'll let you know as soon as we have an update.
Swetha Seelam 0 Reputation points Microsoft Employee

2024-04-02T13:22:07.96+00:00

@Smaran Thoomu Any update on this?
Smaran Thoomu 34,960 Reputation points Microsoft External Staff Moderator

2024-04-03T10:09:42.0366667+00:00

@Swetha Seelam I'm still waiting to hear back from my internal team. If I receive any updates, I'll make sure to let you know.
Smaran Thoomu 34,960 Reputation points Microsoft External Staff Moderator

2024-04-09T09:24:03.1466667+00:00

@Swetha Seelam We haven't received any update from the internal team. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?