A pipeline in Azure Data Factory loses certain fields, yet a newly created pipeline with the same setup does not.

Question

In Azure Data Factory, a pipeline consists of two activities: one is a Web Activity and the other is a Copy Data Activity. The pipeline first acquires a token through the Web Activity and then uses the token to retrieve data from the Synapse database. However, for about a week, two fields of data have been completely lost in this pipeline, namely managerId and managerName. After recreating the pipeline, the user found that all information is visible in the newly created pipeline without any field loss. The configurations of these two pipelines are identical, and the user wants to know why one pipeline is unable to read certain fields.

Answer 1

@Yujin Zhu (Shanghai Wicresoft Co Ltd)

Thanks for reaching out to Microsoft Q&A.

It seems like the issue might be related to the token acquired by the Web Activity.

Token Expiry or Scope: The token used in the problematic pipeline might have expired or has insufficient scope to access the managerId and managerName fields.

Data Source Changes: There might have been changes in the Synapse database schema or permissions that affect the old pipeline but not the new one.

Pipeline Configuration: Even though the configurations appear identical, there might be subtle differences or corruption in the old pipeline’s configuration.

Caching Issues: The old pipeline might be using cached metadata or data, leading to inconsistencies.

You might want to compare the tokens and permissions used by both pipelines and check for any recent changes in the Synapse database.

Hope this helps. Do let us know if you any further queries.

Answer 2

It seems like the issue with the missing managerId and managerName fields could be related to how the pipeline is reading the data from the Synapse database. If the original pipeline was not reading these fields correctly but the new one does, it could be due to factors like:

Metadata cache issue: Sometimes, Azure Data Factory caches metadata and it could be out of sync, causing certain fields to not be read correctly. Recreating the pipeline might have resolved this issue by forcing a fresh metadata read.

Configuration or connection issues: Even though both pipelines have identical configurations, there could be a subtle difference in how they interact with the data source (e.g., the Web Activity or Copy Data Activity settings might slightly differ).

Data inconsistency: Ensure the data in the Synapse database hasn't changed (e.g., schema changes or missing values). The new pipeline might be pulling in data from a different snapshot.

It’s recommended to carefully compare the connection settings, metadata caching, and data source configuration in both pipelines.