HTTP trigger to function fails as some scale-up operation is going on.

Question

Hello,
I have this use case where I am triggering a azure function using ADF and it fails saying internal server error.
It tells me try after sometime, I have put 60 retry but is there better way to do this.
The azure function sets the throughput on cosmosDb table programmatically.

Is there any workaround for this ? How can I solve this ?

Error Details on ADF:

Failure type: User configuration issue
Details:
Call to provided Azure function 'CosmosDBHouseKeeping' failed with status-'InternalServerError' while invoking 'POST' on 'https://fa-tgs-prod.azurewebsites.net' and message - 'Invoking Azure function failed with HttpStatusCode - InternalServerError.'.

Error Logs below:

Blockquote

Exception while executing function: CosmosDBHouseKeeping One or more errors occurred. (ERROR: Unable to set throughput of 4000 for DlpDB.DlpAppPerUser_20221018. Message: {"Errors":["There is another scale-up operation which is currently in progress. Please retry after sometime."]}
ActivityId: d770ec61-b6fb-4d0e-8638-c4875ef6c6dc, Request URI: /apps/fbca2fc9-d702-4182-b791-1269083fc36e/services/5d2d9253-530a-4d71-b262-99fe7adf1b88/partitions/31aba90d-265a-4985-ae20-48b647e6603d/replicas/133075347187966197p, RequestStats:
RequestStartTime: 2022-10-19T12:06:32.7680511Z, RequestEndTime: 2022-10-19T12:06:32.7680511Z, Number of regions attempted:1
{"systemHistory":[{"dateUtc":"2022-10-19T12:05:37.9180624Z","cpu":0.244,"memory":631436220.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0413,"availableThreads":32761,"minThreads":64,"maxThreads":32767},"numberOfOpenTcpConnection":246},{"dateUtc":"2022-10-19T12:05:47.9280566Z","cpu":0.454,"memory":631309100.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0177,"availableThreads":32764,"minThreads":64,"maxThreads":32767},"numberOfOpenTcpConnection":253},{"dateUtc":"2022-10-19T12:05:57.9380484Z","cpu":0.678,"memory":631282400.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0094,"availableThreads":32762,"minThreads":64,"maxThreads":32767},"numberOfOpenTcpConnection":255},{"dateUtc":"2022-10-19T12:06:07.9480435Z","cpu":0.613,"memory":631093556.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.018,"availableThreads":32762,"minThreads":64,"maxThreads":32767},"numberOfOpenTcpConnection":255},{"dateUtc":"2022-10-19T12:06:17.9580442Z","cpu":0.617,"memory":630986044.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.017,"availableThreads":32764,"minThreads":64,"maxThreads":32767},"numberOfOpenTcpConnection":255},{"dateUtc":"2022-10-19T12:06:27.9680540Z","cpu":0.546,"memory":630670048.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0153,"availableThreads":32762,"minThreads":64,"maxThreads":32767},"numberOfOpenTcpConnection":255}]}

Blockquote

Answer 1

500 internal server error. I have found that as another operation was going on and it require around 18 gb of data to be processed. This operation is not able to carry out. When I increased the throughput to 400000 of cosmos, then this function is successful.