How to Transform files in subfolders with one script in databricks

reddy 41

i have a adls gen2 folder with sub folders with parquet files in each folder. My requirement is to transform all parquet files in sub folders and load into another folder in adls gen 2 with same folder structure with one script. is it possible to do or do i need multiple scripts?

Can we do this in databricks or adf?
Please suggest..
Thanks

HarithaMaddi-MSFT 10,136 Reputation points

2020-08-19T06:06:07.783+00:00

Hi @reddy ,

If your query is addressed, request you to mark the answer that helped you as “Mark as Answer” and "Up-Vote" so that other forum members can benefit from it. If you need any further help do let us know.

1 answer

HarithaMaddi-MSFT 10,136 Reputation points

2020-08-17T11:59:49.603+00:00

Hi @reddy ,

Welcome to Microsoft Q&A Platform.

This can be implemented in Azure databricks and Azure data factory as well. There is no straight forward approach in databricks(Need to use SDKs), hence data factory is feasible for this as per my understanding.

Below is one approach in data factory: we can use "Get metadata" activity to get details of the files in the particular folder structure and they can be passed to foreach activity. Inside foreach activity, having dataflow will give us possibilities to apply many transformations as shown below.

Hope this helps! Please let us know if our understanding is incorrect or for further queries and we will be glad to assist further.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members
Please sign in to rate this answer.
reddy 41 Reputation points

2020-08-18T02:41:16.147+00:00

hi @HarithaMaddi-MSFT ,
here, i am using databricks notebook activity for transforming the parquet files. so Can for each activity against the databricks notebook?
I have the transformation logic in databricks notebook which i want to apply for each file in the folder.

HarithaMaddi-MSFT 10,136 Reputation points

2020-08-18T05:05:47.33+00:00

@reddy - Yes, databricks notebook can be executed inside "Foreach activity" using "Notebook activity". If you pass filename as parameter to the databricks notebook, that will be executed against each file in the loop and it can be set as sequential or parallel (with settings of batch count as well for parallel mode). Please check below documentation for more details.
transform-data-using-databricks-notebook

Hope this helps! Please let us know for further queries.

HarithaMaddi-MSFT 10,136 Reputation points

2020-08-21T15:37:42.363+00:00

Hi @reddy , we have not heard back from you. Please let us know for more queries and we will be glad to assist further.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to Transform files in subfolders with one script in databricks

1 answer

Your answer