data flow drops column in different data factory

Peter McLarty 1

I have a data flow which is not displaying a column on the last step the sink when I view the the data in Date Preview.
The source has been pushed from for our repository. Checking the data flow JSON matches between the Dev data factory and the production one, suggests all is the same.
It's a simple data flow, read a JSON file in blob, filter some columns, complete a transform and then land it in Azure SQL

It doesn't raise any errors, but something is failing

How to debug if any method? Is this an issue with the factory?

Plenty of others are working fine in this same set up.

HarithaMaddi-MSFT 10,136 Reputation points

2020-09-14T11:45:41.62+00:00

Hi @Peter McLarty ,

Welcome to Microsoft Q&A Platform.

Sorry that you are experiencing the issue. Can you please confirm if table in SQL Database in production environment has same structure as development environment. Also, can you please check and confirm if issue persists by importing the schema in Sink Dataset from connection. Please do share dataflow JSON as text file if possible and share pipeline run id, region of data factory if column is getting dropped after you run the pipeline as well to proceed with further investigation.
HarithaMaddi-MSFT 10,136 Reputation points

2020-09-15T08:11:56.887+00:00

Hi @Peter McLarty ,

We have not received a response from you. Are you still facing the issue? If you found a solution, would you please share it here with the community? Otherwise, let us know details and we will continue to engage with you on the issue.
HarithaMaddi-MSFT 10,136 Reputation points

2020-09-16T08:41:30.57+00:00

Hi @Peter McLarty ,

We still have not heard back from you. Following up to check if you found a solution, would you please share it here with the community? Otherwise, let us know details and we will continue to engage with you on the issue.

Share via

data flow drops column in different data factory