information required

Question

Niren Adhikary 96

Hello,

1.What criteria we need to consider while choosing between Azure Synapse pipeline and ADF ?

2.Can we convert the file format to parquet format before loading/ingestion into ADLS ?

3.What should we choose for data transformation requirement like aggregation/cleanup etc and why ?

4.How do we convert parquet to delta format ?

Thank you

1 answer

Answer 1

Samy Abdul 3,376

Hi @Niren Adhikary , yes you can convert the file in to Parquet format by defining

sink dataset as parquet , for the data transformations ADF has Dataflows which is

widely used , if you are not doing transformations in Databricks. Thanks

Niren Adhikary 96 Reputation points

2022-12-07T07:51:01.673+00:00

@Samy Abdul Thanks for reply. We also have requirement of data transformation as mentioned in the 3rd query. Basically I have 4 queries but the answer is incomplete.
Samy Abdul 3,376 Reputation points

2022-12-07T08:05:05.143+00:00

Hi , for that mostly Databricks being used, below videos nicely explains both parquet to

delta tables as well as CSV to parquet.

https://www.youtube.com/watch?v=eCIHO7qXdv4

https://www.youtube.com/watch?v=RLXzcAMTnLY

For data validation in dataflow
https://www.youtube.com/watch?v=ozhHsto-Dqg Thanks
Bhargava-MSFT 31,356 Reputation points Microsoft Employee Moderator

2022-12-12T22:50:20.8+00:00

Thank you, @Samy Abdul

Hi @Niren Adhikary ,
I am checking to see if the above answer from @Samy Abdul helped here.

If this answers your question, please consider accepting the answer by hitting the Accept answer button, as it helps the community.