Create an ETL pipeline from CosmosDB (JSON structure) to blob storage creating partitions

Aristeides Douvaras 1

Hi everyone,

I am trying to create an ETL in ADF.
Specifically, i try to ingest data on a recurrent basis (hourly jobs) from CosmosDB to blob storage.

Currently, I am using the "copy data" feature on ADF but i think that i need to use another feature of ADF cause i cannot find how i can create partitions using only the "copy data" feature.

The partitions should be created from the datetime column of the data.
The data on cosmosDB is of Json format.

Any ideas?

Saurabh Sharma 23,791 Reputation points Microsoft Employee

2021-12-04T01:38:15.59+00:00

Hi @Aristeides Douvaras ,

Thanks for using Microsoft Q&A!!
What do you mean by partitioning at storage end? Can you please share sample json data against which you want the partitions to be created?

Thanks
Saurabh
Saurabh Sharma 23,791 Reputation points Microsoft Employee

2021-12-08T01:48:22.81+00:00

Hi @Aristeides Douvaras ,

We haven't heard back from you. Just wanted to check if you are you still facing the issue? In case If you already found a solution, would you please share it here with the community? Otherwise, let us know and we will continue to engage with you on the issue.

Thanks
Saurabh
Aristeides Douvaras 1 Reputation point

2021-12-08T10:30:03.167+00:00
Hi @Saurabh Sharma ,

Thank you for your time!

Actually, what i wanted to do is that i wanted to create different folders for every hour of the day for every different day in the blob storage.
The hierarchy for should be something like:

20211208
-- 0
-- 1
-- 2
-- etc. (up to 23)

20211209
-- 0
-- 1
-- 2
-- up to 23

I started working with the copy ingest function of ADF but i figured out that by using databricks this is much easier. The thing is that i believe that it is maybe an overkill to use Databricks as you need to have a cluster running.

As far as i understood in order to create this structure using the copy ingest function of ADF is very complicated.
Do you have something to propose?

The structure of the JSON (which is the source ) doesnt matter it just includes a datetime column which is the one i want to use in order to create the folder structure i mentioned above.

Share via

Create an ETL pipeline from CosmosDB (JSON structure) to blob storage creating partitions