question

KonradI-6199 avatar image
0 Votes"
KonradI-6199 asked OuryBa-MSFT answered

Migrating data to Cosmos DB is too slow

Hello, i have a question regarding loading data to Cosmos DB database.
In our scenario, we had a PostgreSQL database and an app that exports three SQL views into CSV files. The total size of those CSV files is about 5 GB. Our colleague needs those files to perform some analytics in PowerBI app.
We generated those files every night and uploaded them into Azure Storage. Then, our colleague downloads those files on his computer and performs his work.
We wanted to use CosmosDB to load data from CSV files and then that data would automatically migrate do Synapse Analytics using Synapse Link.
However, i tried three ways of loading data into Cosmos DB and all of them seem to work very slowly.
First method - loading data using Azure Data Factory pipeline - it worked for about six hour per file.
Second method - i wrote an console app based on this code:
https://github.com/Azure-Samples/cosmos-dotnet-bulk-import-throughput-optimizer
I added reading data from my CSV file and run it, it worked for about six hour for one file.

Third method - i wrote a console app based on this code:
https://github.com/Azure/azure-cosmosdb-bulkexecutor-dotnet-getting-started
This one was the fastest, it loaded one CSV file in about 4 hours and 40 minutes, but it's still slow.
Is there any way to make it faster? Would you recommend us any other way of loading data into Cosmos DB, that would work faster?

PS our Cosmos DB account runs in serverless mode.

azure-cosmos-db
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

OuryBa-MSFT avatar image
0 Votes"
OuryBa-MSFT answered

@KonradI-6199 Thank you for posting your query on Microsoft Q&A and for using Azure services.

My understanding is that you are looking for a faster way in migrating to cosmos Db. as mentioned above you have tried different options but still take long time to even upload one file.

Have you thought of increasing the RUs in Cosmos DB that would help in getting more throughput?

Regards,
Oury

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@KonradI-6199 Checking if you were able to see the above reply I provided.

Regards,
Oury

0 Votes 0 ·