Create an ETL pipeline from CosmosDB (JSON structure) to blob storage creating partitions
I am trying to create an ETL in ADF.
Specifically, i try to ingest data on a recurrent basis (hourly jobs) from CosmosDB to blob storage.
Currently, I am using the "copy data" feature on ADF but i think that i need to use another feature of ADF cause i cannot find how i can create partitions using only the "copy data" feature.
The partitions should be created from the datetime column of the data.
The data on cosmosDB is of Json format.
Hi @Saurabh Sharma ,
Thank you for your time!
Actually, what i wanted to do is that i wanted to create different folders for every hour of the day for every different day in the blob storage.
The hierarchy for should be something like:
-- etc. (up to 23)
-- up to 23
I started working with the copy ingest function of ADF but i figured out that by using databricks this is much easier. The thing is that i believe that it is maybe an overkill to use Databricks as you need to have a cluster running.
As far as i understood in order to create this structure using the copy ingest function of ADF is very complicated.
Do you have something to propose?
The structure of the JSON (which is the source ) doesnt matter it just includes a datetime column which is the one i want to use in order to create the folder structure i mentioned above.
Sign in to comment