Why has Synapse Analytics pipeline reverted to a previous version of a notebook?

Daniella Quinn 0

I made some changes to a couple of Synapse Analytics notebooks on Friday. These notebooks are part of an existing pipeline. I committed the changes, PRd and merged the branch as normal etc.

The pipeline runs daily. It ran perfectly fine on Saturday, with the changes I'd made the day before. It also ran fine on Sunday. When I checked this morning's run (Monday) it had also run successfully, but the notebooks it ran were a previous version of the notebooks before my PR merge on the Friday. Saturday's and Sunday's run included the updated notebooks, but Monday's had reverted back?

I ran it manually just now, and it's run as normal with the updates again.

Very confused.

Harishga 5,270 Reputation points Microsoft Vendor

2024-06-17T12:48:07.5166667+00:00

Hi @Daniella Quinn
Welcome to Microsoft Q&A platform and thanks for posting your question here.

It seems like your Synapse Analytics pipeline is running different versions of your notebooks inconsistently. This is unusual because the pipeline ran the updated notebooks correctly on Saturday and Sunday but reverted to an older version on Monday.

To troubleshoot this issue, you can check if there's a caching mechanism causing the pipeline to run the older version of the notebooks. You can also look at the trigger configuration of your pipeline to see if it's pointing to a different branch or commit. Using Synapse Studio to monitor your pipeline runs can help you identify any discrepancies in the execution details between the runs on different days.

Make sure that the pipeline is properly versioned and that the correct version is being triggered. It's possible that a different version of the pipeline, which includes the older notebooks, was inadvertently triggered. You can compare the configuration and logs of the manual run with the scheduled run to see if there are any differences.

To resolve the issue, you can try re-publishing your pipeline to ensure that the latest changes are picked up. Clearing any caches or temporary files that might be causing the older version to run can also help. Reviewing the execution logs for the Monday run to see if there are any anomalies or errors reported can provide additional information.

Finally, validating the PR merge to confirm that the changes were merged into the correct branch and that the pipeline is pointing to this branch can help fix the issue.

Reference
https://learn.microsoft.com/en-us/azure/synapse-analytics/monitoring/how-to-monitor-pipeline-runs

I hope this information helps you. Let me know if you have any further questions or concerns.
Harishga 5,270 Reputation points Microsoft Vendor

2024-06-18T09:04:57.4233333+00:00

@Daniella Quinn
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Share via

Why has Synapse Analytics pipeline reverted to a previous version of a notebook?