Share via

How to implement a delta connector in a ADF dataflow using Azure .Net SDK (C#)

Divya Rani Pallapothu 21 Reputation points
2021-02-10T17:50:26.147+00:00

I have been trying to create a dataflow from AzureSqlDatabase to Delta Format on ADLS Gen2. We have a Sink Type = delta option in the ADF Dataflow UI which enables the direct ingestion instead of an intermediate parquet conversion step / staged copy.

How can we create the dataflow with sink type = delta? Can we implement this delta connector using Azure .net SDK(C#)?

66619-image.png

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
0 comments
Answer accepted by question author
  1. Saurabh Sharma 23,866 Reputation points Microsoft Employee Moderator
    2021-02-12T01:34:10.917+00:00

    @Divya Rani Pallapothu Thanks for using Microsoft Q&A !!
    Yes you can use Delta as a Sink from UI and here is the documentation and a video tutorial for the same. As far as I know you cannot however Delta as a sink in .NET SDK. I see, DataFlowSink class seems using only Datasets while creation. Please provide this as a feedback at Uservoice so that community can upvote and the products team can plan to include it in future releases.

    ----------

    Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Divya Rani Pallapothu 21 Reputation points
    2021-02-19T04:34:01.55+00:00

    Thanks Saurabh! I could create Delta as a Sink from UI. Also, figured out a way to implement the same via .NET SDK.

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.