Python function to upload files to storage failing with 403 TooManyDevicesError

Question

I have a python app running on an embedded linux device. The app runs async code that connects an iothub device client. I have a class called device_client_wrapper that uses a semaphore to ensure that only one function calls device client methods at a time, as that was causing me issues before.

I just added a new function to this class which is used to upload files from the device to azure blob storage. It was working fine for a while, and often works the first time, but then if I execute it too many times I get the following error:

ERROR upload_file Exception: Unexpected failure
ERROR Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 33, in handle_result
    return await callback.completion()
  File "/usr/lib/python3.10/site-packages/azure/iot/device/common/async_adapter.py", line 91, in completion
    return await self.future
azure.iot.device.exceptions.ServiceError: HTTP operation returned: 403 TooManyDevicesError(Error: Forbidden)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/cogo_ig60/cloud/device_client_wrapper.py", line 99, in upload_file
    blob_info = await self.device_client.get_storage_info_for_blob(blob_name)
  File "/usr/lib/python3.10/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 571, in get_storage_info_for_blob
    storage_info = await handle_result(callback)
  File "/usr/lib/python3.10/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 57, in handle_result
    raise exceptions.ClientError("Unexpected failure") from e
azure.iot.device.exceptions.ClientError: Unexpected failure

This is the function causing the error:

async def upload_file(self, path_to_file: str, blob_name: str):
    """Upload a file to Azure Blob Storage."""
    try:
        async with self.semaphore:
            blob_info = await self.device_client.get_storage_info_for_blob(blob_name)
            sas_url = "https://{}/{}/{}{}".format(
                blob_info["hostName"],
                blob_info["containerName"], 
                blob_info["blobName"],
                blob_info["sasToken"]
            )
            logger.info(
                f"Uploading file {path_to_file} to blob storage as {blob_name}")
            with BlobClient.from_blob_url(sas_url) as blob_client:
                with open(path_to_file, "rb") as f:
                    result = blob_client.upload_blob(f, overwrite=True)
                    return (True, result)
    except FileNotFoundError as e:
        # catch file not found and add an HTTP status code to return in notification to IoT Hub
        logger.error(f'upload_file FileNotFoundError: {e}')
        logger.error(traceback.format_exc())
        e.status_code = 404
        return (False, e)
    except AzureError as e:
        # catch Azure errors that might result from the upload operation
        logger.error(f'upload_file AzureError: {e}')
        logger.error(traceback.format_exc())
        return (False, e)
    except Exception as e:
        # catch all other errors
        logger.error(f'upload_file Exception: {e}')
        logger.error(traceback.format_exc())
        return (False, e)

It seems to be an issue with device_client.get_storage_info_for_blob. I know there are a max of 10 concurrent file uploads per device, but I am uploading 7 fiiles, then they finish uploading successfully before I try again. Even so, with the semaphore it should prevent any concurrent file uploads. That is unless I'm missing something about how my code is working.

Is something not being released properly, or not able to complete properly?

This is python 3.10.4 btw.

Answer

Hello @Lachlan Pedersen ,

thank you for reaching out to this community.

Does it seem you are using the Azure IoT Device Client SDK for Python, available as open source on GitHub? In that case, I propose to open an issue there. The GitHub repo is maintained by the product team so you get support from the team itself.

Do you support sending a '200' acknowledgment once a file is sent successfully? See step 8 in this documentation. You need to pass the correlation ID of the file being uploaded. This is to release this upload.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

Share via

Python function to upload files to storage failing with 403 TooManyDevicesError

1 answer