ServerError; BlobDownloadMiscError; Miscellaneous error encountered while downloading one of the specified Azure Blob(s)

Farella, Martha 1 Reputation point
2022-08-25T19:04:37.573+00:00

I'm running 6 tasks on a single job. All tasks use the same resource file (with an SAS token). Tasks fail about 50% of time and give the following error:

Failure info
task1_huc020301030703_time2022-08-25-18-28-46
Category: ServerError
Code: BlobDownloadMiscError
Message: Miscellaneous error encountered while downloading one of the specified Azure Blob(s)

Name: BlobSource
Value:
https://insightmlstoragesbx.blob.core.windows.net/flood-ml-code/src/deployment/local_deployment/deploy_local.cfg

Name: FilePath
Value:
/mnt/batch/tasks/workitems/Data_Processing_Notebook/job-1/task1_huc020301030703_time2022-08-25-18-28-46/wd/src/deployment/local_deployment/deploy_local.cfg

Name: Message
Value:
FileIOException('Failed to allocate 382 bytes for file /mnt/batch/tasks/workitems/Data_Processing_Notebook/job-1/task1_huc020301030703_time2022-08-25-18-28-46/wd/src/deployment/local_deployment/deploy_local.cfg')

So, if I submit 6 tasks 3 of them ultimately fail and the other 3 will run. Meaning that the resource_file directory is correct, since 3 of the tasks that are using the same directory as failed tasks work.

I have confirmed that all tasks are being run on separate nodes.

Tasks fail within seconds of being added, so the SAS token is still valid

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
1,528 questions
Azure Batch
Azure Batch
An Azure service that provides cloud-scale job scheduling and compute management.
249 questions