Query data when recreating Cosmos Sql collections in data factory data flow

Question

Query data when recreating Cosmos Sql collections in data factory data flow

Wentao Ding 26 Microsoft Employee

We need to wipe out the old data and get fresh data from the data source every day. In the data flow, for cosmos sql sink, there is an option to Recreate the collection. But while it is writing the rows, it is possible that query a certain row that has not been written to the cosmos sql, since it takes about 25 mins to complete the pipeline.

So I was wondering is there a better way to handle this scenario? like populate the data to a staging table and swap?

0 comments

1 answer

Your answer

Answer 1

HimanshuSinha 19,637 Microsoft Employee Moderator

Hello @Wentao Ding ,
Thanks for the ask and using the Microsoft Q&A platform .

it is possible that query a certain row that has not been written to the table

I am assuminmg that the table referenced here is on the source side and if thats the case then yes you are write a query and MDF will execute the query to pull the data .

Please do let me know how it goes .
Thanks
Himanshu
Please do consider clicking on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

Wentao Ding 26 Reputation points Microsoft Employee

2021-08-13T22:52:53.05+00:00

Hi Himanshu,

Thanks for your reply.

nope, the table here is the sink table. Since I selected the option to Recreate the collections, It looks it will drop the old one and then start writing new rows, but it will take about 25 mins to complete. Then during this time window, if there is only partial data, the request would fail. I was wondering if there is a way to swap collections. (For example, populate the data to a temp/staging table, and swap to prod)

Wentao Ding
HimanshuSinha 19,637 Reputation points Microsoft Employee Moderator

2021-08-19T20:36:32.34+00:00

Hello @Wentao Ding ,

Then during this time window, if there is only partial data, the request would fail.

I am assuming you meant partial data on the source side ,let me know if thats accurate .
If thats the case , you can do the copy activity in two hops , you can copy the data first to a blob and then from the blob to the COSMOSDB .

Please do let me know how it goes .
Thanks
Himanshu
HimanshuSinha 19,637 Reputation points Microsoft Employee Moderator

2021-08-23T19:39:12.897+00:00

Hello @Wentao Ding ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

Share via

Query data when recreating Cosmos Sql collections in data factory data flow

1 answer

Your answer