How to implement a delta connector in a ADF dataflow using Azure .Net SDK (C#)

Divya Rani Pallapothu 21 Reputation points
2021-02-10T17:50:26.147+00:00

I have been trying to create a dataflow from AzureSqlDatabase to Delta Format on ADLS Gen2. We have a Sink Type = delta option in the ADF Dataflow UI which enables the direct ingestion instead of an intermediate parquet conversion step / staged copy.

How can we create the dataflow with sink type = delta? Can we implement this delta connector using Azure .net SDK(C#)?

66619-image.png

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,895 questions
0 comments No comments
{count} votes

Accepted answer
  1. Saurabh Sharma 23,771 Reputation points Microsoft Employee
    2021-02-12T01:34:10.917+00:00

    @Divya Rani Pallapothu Thanks for using Microsoft Q&A !!
    Yes you can use Delta as a Sink from UI and here is the documentation and a video tutorial for the same. As far as I know you cannot however Delta as a sink in .NET SDK. I see, DataFlowSink class seems using only Datasets while creation. Please provide this as a feedback at Uservoice so that community can upvote and the products team can plan to include it in future releases.

    ----------

    Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.


1 additional answer

Sort by: Most helpful
  1. Divya Rani Pallapothu 21 Reputation points
    2021-02-19T04:34:01.55+00:00

    Thanks Saurabh! I could create Delta as a Sink from UI. Also, figured out a way to implement the same via .NET SDK.