delta sink inserts blank row

arkiboys 9,701 Reputation points
2022-07-05T21:45:22.363+00:00

Hi,
I am investigating why the delta sink transformation is inserting blank row.
After the dataflow is run, I then read the delta parquet files in the storage and shows blank row has been inserted as well as other valid rows.
Any thoughts?
Thank you

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,209 questions
0 comments No comments
{count} votes

Accepted answer
  1. HimanshuSinha-msft 19,476 Reputation points Microsoft Employee
    2022-07-06T21:00:49.177+00:00

    Hello @arkiboys ,
    Thanks for the question and using MS Q&A platform.

    As we understand the ask here is why there is a blank record in the paraquet file , please do let us know if its not accurate.
    The UPSERT is always set on some key column . I have not seen the data ( which should have been supper helpful) , but i was wondering how is it possible that you have rows which are all blank .

    Anyways I think to debug the case , just do the oppsite in the filter , try to how many rows are there with

    length(trim(venture_number) = 0

    This should help to find the rows like that and then we can think as to what can be done .
    Please do let me if you have any queries.
    Thanks
    Himanshu


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.