Calling Microsoft Defender API from data factory

Question

Hi All, 

I have a web activity that calls Microsoft Defender API (https://api-uk.securitycenter.microsoft.com/api/machines/SoftwareInventoryExport?sasValidHours=24). It's authenticated with a bearer token from an earlier activity. The returned results should be a list of urls to files to be downloaded.

This activity works in DEV and PRE-PROD (not many files to download), however in PROD it consistently gives an error: 

"Error calling the endpoint 'https://api-uk.securitycenter.microsoft.com'. Response status code: 'ClientSideException HttpStatusCode : 408, HttpStatusString: RequestTimeout'. More details: Exception message: 'NA - Unknown [ClientSideException] A task was canceled.'. Request didn't reach the server from the client. This could happen because of an underlying issue such as network connectivity, a DNS failure, a server certificate validation or a timeout. Url endpoint request timed out. Please make sure the endpoint response is within 1 minute and retry."

 

Anybody came across this before? It is not an authentication issue. To me it seems it just doesn't compile the results in 1 minute to return, hence the server closes the connection. But if that's the case, unsure how to solve that.

 

Just to note, the one time it actually returned a results it was a list of about 1,400 URLs. 

Thank you.

Answer 1

The Azure Data Factory's web activity has a default timeout of 1 minute, and it seems like the Microsoft Defender API is taking longer than that to compile the results in your PROD environment.

You can increase the timeout of the web activity. In the policy section of the web activity, set timeout to a larger value (in seconds). However, please note that the maximum timeout period for a web activity in Azure Data Factory is 230 seconds (a bit less than 4 minutes). If you think the API response will take longer than that, you may need to consider other approaches. If the API supports pagination, you could make multiple calls to the API, each time retrieving a subset of the data, rather than trying to retrieve everything at once. However, this will depend on whether the Microsoft Defender API supports pagination.