Azure Data Factory

Question

Azure Data Factory

Imran Mondal 246

EveryDay My Data Bricks job dumping data in YYYY/MM/DD/HH format folder in Blob Storage
like this -

hourlydata/2021/03/10/00/part-00000-tid-8b61-4d85-a6cb-249245ccad76-39980-1-c000.csv
hourlydata/2021/03/10/01/part-00000-tid-9c81-dc555aaa0f8-41519-1-8c3-46ca-97-c000.csv
hourlydata/2021/03/10/02/part-00000-tid-9c81-dc555aaa0f8-41519-1-8c3-46ca-97-c000.csv

I have a data set that resides under a folder path where the date is dynamic, inside each hour folder along with CSV files, we have other files also. But I want to select only the CSV file.

As soon as my Databricks Jobs dumps data every hour, I would like to read those new data using data factory and load it into a database, please suggest

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2021-03-17T20:28:37.573+00:00

Hi @Imran Mondal ,

Just checking in to see if the below suggestion from anonymous userChaudhari was helpful. If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread.

And, if you have any further query do let us know.

Thanks
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2021-03-18T18:51:00.307+00:00

Hi @Imran Mondal ,

We still have not heard back from you. Just wanted to check if you still need assistance on this query? Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

And if you have further query, do let us know.

1 answer

Your answer

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2021-03-17T20:28:37.573+00:00

Hi @Imran Mondal ,

Just checking in to see if the below suggestion from anonymous userChaudhari was helpful. If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread.

And, if you have any further query do let us know.

Thanks
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2021-03-18T18:51:00.307+00:00

Hi @Imran Mondal ,

We still have not heard back from you. Just wanted to check if you still need assistance on this query? Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

And if you have further query, do let us know.

Answer 1

Vaibhav Chaudhari 38,916 Volunteer Moderator

Try using Event based triggers in ADF. You should configure it such way that as soon as new file is created ending with .csv, pipeline will use this file to copy/process

See point 6 in below doc

https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger

----------

Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav

Imran Mondal 246 Reputation points

2021-03-15T08:25:10.983+00:00

So, My data bricks are job creating this year, month, Day, hour folder dynamically when it runs,

So these triggers in ADF will work in these scenarios also, like every hour when Data dump happens it will start the data Factory job.
Vaibhav Chaudhari 38,916 Reputation points Volunteer Moderator

2021-03-15T10:33:48.343+00:00

Yes, as soon as file is created or modified, based on these events, ADF pipeline will get triggered.. You have to configure the event trigger as per your requirement

See step by step tutorial - https://www.mssqltips.com/sqlservertip/6063/create-event-based-trigger-in-azure-data-factory/

Share via

Azure Data Factory

1 answer

Your answer