Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
Azure Data Flow insert null even there is value
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 34%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
Riz
0
Reputation points
Hello Community,
I am looking for your help and its priority. I am trying to import file using inline source type from Common Data Model. F&O Dynamic 365 application export the files in data lake and with the help of azure data flow I am importing the customer.csv file using Manifest file.
Just for little intro Manifest file provides the details of columns and its generated from Dynamic 365 F&O application
The problem here is in customer.csv I am seeing value for the 2 columns. There are only 2 rows out of millions rows where the values are populated in those 2 columns - Column A and Column B. Those 2 records are at the middle of the file. Column A is datetime and column B is int data type in file.
The field names are reading correctly at the sink but value is generated as null for those 2 columns. I checked the schema of file for those 2 columns - Column A and Column B. Its populated correctly ( ref image 2). Also change data type in schema options (ref image 3) still doesn't work.
If I manipulated the source file customer.csv by adding values in the 1st row in file for those 2 columns . It generates the output in the sink. Why is that so in Data flow? How do I handle the records if its appear in the middle of the file?
Please help !!
Image 1:
Shows the connection reading from Common Data Model
Image 2:
It shows correctly the data type of those 2 columns. I named them as columnA and ColumnB
Image 3:
I set the schema for both using schema options for date and time both
Best Riz
Azure Data Lake Storage
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
{count} votes
-
AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator2023-04-10T17:12:01.2566667+00:00 Hi Riz , Welcome to Microsoft Q&A platform and thanks for posting your question here.
Your ask is slightly confusing. I am not able to understand it completely, Kindly help with below deails.
Could you please explain little more, what you mean, when you say "There are only 2 rows out of millions rows where the values are populated in those 2 columns - Column A and Column B. Those 2 records are at the middle of the file". - Do you mean , all other rows are coming as NULL , but only 2 records are coming correctly in the source dataset?
Also, what you mean, when you say "If I manipulated the source file customer.csv by adding values in the 1st row in file for those 2 columns . It generates the output in the sink.". What exactly you did here?
Could you please check what is the row delimiter you have set in the csv dataset . If it's
\\r\\n, could you please change it to\\nand see if it's working well? as sometimes multiline characters create issue in rowdelimiter for .csv file.Thankyou. Awaiting your response.
-
AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator
2023-04-13T07:27:15.8166667+00:00 Riz , Just checking if you got a chance to visit the previous response?
-
AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator
2023-04-19T04:19:04.9566667+00:00 Riz , Just checking if your issue has been resolved? Kindly share the resolution details with the community else revert back with the required details. Thankyou
Sign in to comment
Sign in to answer