Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,142 questions
Request size too large - CosmosDB
Ferrazza, Paolo {DSRM~Rotkreuz-tro}
1
Reputation point
I'm trying to copy from one cosmosDB container to another one. I've tried both the copy activity and to write a custom dataflow, in both situations I get error messages saying that request size is too large with sizes like this "2208232". So it's just a little bit above 2MB. Notice that in our source cosmosDB items have size very close to 2MB so it's like there is some sort of bug in ADF which is adding some size to the message. I've also set write batch size = 1 and it keeps failing.
Example of copy activity configuration:
As you can see there is no transformation, just moving items to another container, so if the item is of the correct size in the source container it should not create errors in the destination container...or am I missing something?
{count} votes
-
AnnuKumari-MSFT 23,271 Reputation points Microsoft Employee2022-11-15T14:05:39.92+00:00 Hi @Ferrazza, Paolo {DSRM~Rotkreuz-tro} ,
Welcome to Microsoft Q&A platform and thanks for posting your question here.
As I understand your issue, while trying to move data from one cosmosDB container to another, it's throwing error 'Request size too large - CosmosDB'. Could you please share the exact size of the source when you say 'Notice that in our source cosmosDB items have size very close to 2MB' , what is the exact size here. Also, what is the cosmosDB API here?
As per MS documentation, Cosmos DB limits single request's size to 2MB. The formula is Request Size = Single Document Size * Write Batch Size.
The workaround suggested if it exceeds this is to split the document into multiple partitions, after that use the For Each activity with a Copy Activity to write every file into Cosmos DB .
Here are few resources you can consider referring to:
https://stackoverflow.com/questions/58653153/what-is-the-size-limit-of-a-single-document-stored-in-azure-cosmos-db
https://learn.microsoft.com/en-us/answers/questions/69129/copy-from-cosmosdb-to-cosmosdb-error-amp34request.htmlPlease let us know how it goes. Thankyou
-
Ferrazza, Paolo {DSRM~Rotkreuz-tro} 1 Reputation point
2022-11-15T14:20:34.813+00:00 Hello,
thanks for your answer.As you can see from my current configuration I'm moving items without changing them, so what fit the first DB should also fit the destination DB, right? Otherwise, it would mean that the source DB accepted something bigger than 2MB.
Sign in to comment