Upload files from your device to the cloud with Azure IoT Hub (Python)

Article
03/16/2023

This article demonstrates how to file upload capabilities of IoT Hub upload a file to Azure blob storage, using Python.

The Send telemetry from a device to an IoT hub quickstart and Send cloud-to-device messages with IoT Hub articles show the basic device-to-cloud and cloud-to-device messaging functionality of IoT Hub. The Configure Message Routing with IoT Hub tutorial shows a way to reliably store device-to-cloud messages in Microsoft Azure blob storage. However, in some scenarios, you can't easily map the data your devices send into the relatively small device-to-cloud messages that IoT Hub accepts. For example:

Videos
Large files that contain images
Vibration data sampled at high frequency
Some form of pre-processed data.

These files are typically batch processed in the cloud, using tools such as Azure Data Factory or the Hadoop stack. When you need to upload files from a device, you can still use the security and reliability of IoT Hub. This article shows you how.

At the end of this article, you run the Python console app FileUpload.py, which uploads a file to storage using the Python Device SDK.

Note

IoT Hub supports many device platforms and languages (including C, Java, Python, and JavaScript) through Azure IoT device SDKs. Refer to the Azure IoT Developer Center to learn how to connect your device to Azure IoT Hub.

Important

File upload functionality on devices that use X.509 certificate authority (CA) authentication is in public preview, and preview mode must be enabled. It is generally available on devices that use X.509 thumbprint authentication or X.509 certificate attestation with Azure Device Provisioning Service. To learn more about X.509 authentication with IoT Hub, see Supported X.509 certificates.

Prerequisites

An active Azure account. (If you don't have an account, you can create a free account in just a couple of minutes.)
An IoT hub. Create one with the CLI or the Azure portal.
A registered device. Register one in the Azure portal.
Python version 3.7 or later is recommended. Make sure to use the 32-bit or 64-bit installation as required by your setup. When prompted during the installation, make sure to add Python to your platform-specific environment variable.
Port 8883 should be open in your firewall. The device sample in this article uses MQTT protocol, which communicates over port 8883. This port may be blocked in some corporate and educational network environments. For more information and ways to work around this issue, see Connecting to IoT Hub (MQTT).

Associate an Azure Storage account to IoT Hub

To upload files from a device, you must have an Azure Storage account and Azure Blob Storage container associated with your IoT hub. Once you associate the storage account and container with your IoT hub, your IoT hub can provide the elements of a SAS URI when requested by a device. The device can then use these elements to construct the SAS URI that it uses to authenticate with Azure Storage and upload files to the blob container.

To associate an Azure Storage account with your IoT hub:

Under Hub settings, select File upload on the left-pane of your IoT hub.
On the File upload pane, select Azure Storage Container. For this article, it's recommended that your storage account and IoT Hub be located in the same region.
- If you already have a storage account you want to use, select it from the list.
- To create a new storage account, select +Storage account. Provide a name for the storage account and make sure the Location is set to the same region as your IoT hub, then select OK. The new account is created in the same resource group as your IoT hub. When the deployment completes, select the storage account from the list.
After you select the storage account, the Containers pane opens.
On the Containers pane, select the blob container.
- If you already have a blob container you want to use, select it from the list and click Select.
- To create a new blob container, select + Container. Provide a name for the new container. For the purposes of this article, you can leave all other fields at their default. Select Create. When the deployment completes, select the container from the list and click Select.
Back on the File upload pane, make sure that file notifications are set to On. You can leave all other settings at their defaults. Select Save and wait for the settings to complete before moving on to the next section.

For more detailed instructions on how to create an Azure Storage account, see Create a storage account. For more detailed instructions on how to associate a storage account and blob container with an IoT hub, see Configure file uploads using the Azure portal.

Upload a file from a device app

In this section, you create the device app to upload a file to IoT hub.

At your command prompt, run the following command to install the azure-iot-device package. You use this package to coordinate the file upload with your IoT hub.
```
pip install azure-iot-device
```
At your command prompt, run the following command to install the azure.storage.blob package. You use this package to perform the file upload.
```
pip install azure.storage.blob
```
Create a test file that you'll upload to blob storage.
Using a text editor, create a FileUpload.py file in your working folder.

Add the following import statements and variables at the start of the FileUpload.py file.

import os
from azure.iot.device import IoTHubDeviceClient
from azure.core.exceptions import AzureError
from azure.storage.blob import BlobClient

CONNECTION_STRING = "[Device Connection String]"
PATH_TO_FILE = r"[Full path to local file]"

In your file, replace [Device Connection String] with the connection string of your IoT hub device. Replace [Full path to local file] with the path to the test file that you created or any file on your device that you want to upload.

Create a function to upload the file to blob storage:

def store_blob(blob_info, file_name):
    try:
        sas_url = "https://{}/{}/{}{}".format(
            blob_info["hostName"],
            blob_info["containerName"],
            blob_info["blobName"],
            blob_info["sasToken"]
        )

        print("\nUploading file: {} to Azure Storage as blob: {} in container {}\n".format(file_name, blob_info["blobName"], blob_info["containerName"]))

        # Upload the specified file
        with BlobClient.from_blob_url(sas_url) as blob_client:
            with open(file_name, "rb") as f:
                result = blob_client.upload_blob(f, overwrite=True)
                return (True, result)

    except FileNotFoundError as ex:
        # catch file not found and add an HTTP status code to return in notification to IoT Hub
        ex.status_code = 404
        return (False, ex)

    except AzureError as ex:
        # catch Azure errors that might result from the upload operation
        return (False, ex)

This function parses the blob_info structure passed into it to create a URL that it uses to initialize an azure.storage.blob.BlobClient. Then it uploads your file to Azure blob storage using this client.

Add the following code to connect the client and upload the file:

def run_sample(device_client):
    # Connect the client
    device_client.connect()

    # Get the storage info for the blob
    blob_name = os.path.basename(PATH_TO_FILE)
    storage_info = device_client.get_storage_info_for_blob(blob_name)

    # Upload to blob
    success, result = store_blob(storage_info, PATH_TO_FILE)

    if success == True:
        print("Upload succeeded. Result is: \n") 
        print(result)
        print()

        device_client.notify_blob_upload_status(
            storage_info["correlationId"], True, 200, "OK: {}".format(PATH_TO_FILE)
        )

    else :
        # If the upload was not successful, the result is the exception object
        print("Upload failed. Exception is: \n") 
        print(result)
        print()

        device_client.notify_blob_upload_status(
            storage_info["correlationId"], False, result.status_code, str(result)
        )

def main():
    device_client = IoTHubDeviceClient.create_from_connection_string(CONNECTION_STRING)

    try:
        print ("IoT Hub file upload sample, press Ctrl-C to exit")
        run_sample(device_client)
    except KeyboardInterrupt:
        print ("IoTHubDeviceClient sample stopped")
    finally:
        # Graceful exit
        device_client.shutdown()


if __name__ == "__main__":
    main()

This code creates an IoTHubDeviceClient and uses the following APIs to manage the file upload with your IoT hub:

get_storage_info_for_blob gets information from your IoT hub about the linked Storage Account you created previously. This information includes the hostname, container name, blob name, and a SAS token. The storage info is passed to the store_blob function (created in the previous step), so the BlobClient in that function can authenticate with Azure storage. The get_storage_info_for_blob method also returns a correlation_id, which is used in the notify_blob_upload_status method. The correlation_id is IoT Hub's way of marking which blob you're working on.
notify_blob_upload_status notifies IoT Hub of the status of your blob storage operation. You pass it the correlation_id obtained by the get_storage_info_for_blob method. It's used by IoT Hub to notify any service that might be listening for a notification on the status of the file upload task.

Save and close the FileUpload.py file.

Run the application

Now you're ready to run the application.

At a command prompt in your working folder, run the following command:
```
python FileUpload.py
```
The following screenshot shows the output from the FileUpload app:
You can use the portal to view the uploaded file in the storage container you configured:

Next steps

In this article, you learned how to use the file upload feature of IoT Hub to simplify file uploads from devices. You can continue to explore this feature with the following articles:

Learn more about Azure Blob Storage with the following links: