Upload files from a device to the cloud with Azure IoT Hub

Article
08/15/2024

This article demonstrates how to:

Use file upload capabilities of IoT Hub to upload a file to Azure Blob Storage, using an Azure IoT device and service SDKs.
Notify IoT Hub that the file was successfully uploaded and create a backend service to receive file upload notifications from IoT Hub, using the Azure IoT service SDKs.

In some scenarios, you can't easily map the data your devices send into the relatively small device-to-cloud messages that IoT Hub accepts. The file upload capabilities in IoT Hub enable you to move large or complex data to the cloud. For example:

Videos
Large files that contain images
Vibration data sampled at high frequency
Some form of preprocessed data

These files are typically batch processed in the cloud, using tools such as Azure Data Factory or the Hadoop stack. When you need to upload files from a device, you can still use the security and reliability of IoT Hub. This article shows you how.

This article is meant to complement runnable SDK samples that are referenced from within this article.

For more information, see:

Important

File upload functionality on devices that use X.509 certificate authority (CA) authentication is in public preview, and preview mode must be enabled. It is generally available on devices that use X.509 thumbprint authentication or X.509 certificate attestation with Azure Device Provisioning Service. To learn more about X.509 authentication with IoT Hub, see Supported X.509 certificates.

Prerequisites

An IoT hub. Some SDK calls require the IoT Hub primary connection string, so make a note of the connection string.
A registered device. Some SDK calls require the device primary connection string, so make a note of the connection string.
IoT Hub Service Connect permission - To receive file upload notification messages, your backend service needs the Service Connect permission. By default, every IoT Hub is created with a shared access policy named service that grants this permission. For more information, see Connect to an IoT hub.
Configure file upload in your IoT hub by linking an Azure Storage account and Azure Blob Storage container. You can configure these using the Azure portal, Azure CLI, or Azure PowerShell.

Overview

This how-to contains two sections:

Upload a file from a device application
Receive file upload notification in a backend application

Upload a file from a device application

This section describes how to upload a file from a device to an IoT hub using the DeviceClient class in the Azure IoT SDK for .NET.

Follow this procedure to upload a file from a device to IoT hub:

Connect to IoT hub
Get a SAS URI from IoT hub
Upload the file to Azure storage
Notify IoT hub of the file upload status

Connect to the device

Call CreateFromConnectionString to connect to the device. Pass the device primary connection string.

AMQP is the default transport protocol.

static string connectionString = "{device primary connection string}";
deviceClient = DeviceClient.CreateFromConnectionString(connectionString);

Get a SAS URI from IoT hub

Call GetFileUploadSasUriAsync to get file upload details. The SAS URI is used in the next step to upload a file from a device to Blob Storage.

const string filePath = "TestPayload.txt";
using var fileStreamSource = new FileStream(filePath, FileMode.Open);
var fileName = Path.GetFileName(fileStreamSource.Name);
var fileUploadSasUriRequest = new FileUploadSasUriRequest
{
    BlobName = fileName
};

FileUploadSasUriResponse sasUri = await _deviceClient.GetFileUploadSasUriAsync(fileUploadSasUriRequest, System.Threading.CancellationToken cancellationToken = default);
Uri uploadUri = sasUri.GetBlobUri();

Upload a file to Azure storage

To upload a file to Azure storage:

Create a blockBlobClient object, passing a file upload URI.
Use the UploadAsync method to upload a file to Blob Storage, passing the SAS URI. You can optionally add Blob upload options and cancellation token parameters.

The Azure Blob client always uses HTTPS as the protocol to upload the file to Azure Storage.

In this example, BlockBlobClient is passed the SAS URI to create an Azure Storage block Blob client and uploads the file:

var blockBlobClient = new BlockBlobClient(uploadUri);
await blockBlobClient.UploadAsync(fileStreamSource, null, null);

Notify IoT hub of the file upload status

Use CompleteFileUploadAsync to notify IoT hub that the device client completed the upload, passing a FileUploadCompletionNotification object. The IsSuccess flag indicates whether or not the upload was successful. After being notified, IoT hub will release resources associated with the upload (the SAS URI).

If file upload notifications are enabled, IoT hub sends a file upload notification message to backend services that are configured for file upload notification.

var successfulFileUploadCompletionNotification = new FileUploadCompletionNotification
{
    // Mandatory. Must be the same value as the correlation id returned in the sas uri response
    CorrelationId = sasUri.CorrelationId,

    // Mandatory. Will be present when service client receives this file upload notification
    IsSuccess = true,

    // Optional, user defined status code. Will be present when service client receives this file upload notification
    StatusCode = 200,

    // Optional, user-defined status description. Will be present when service client receives this file upload notification
    StatusDescription = "Success"
};

await _deviceClient.CompleteFileUploadAsync(successfulFileUploadCompletionNotification);

SDK file upload sample

The SDK includes this file upload sample.

Receive a file upload notification in a backend application

You can create a backend service to receive file upload notification messages from IoT hub.

The ServiceClient class contains methods that services can use to receive file upload notifications.

To receive file upload notification:

Call CreateFromConnectionString to connect to IoT hub. Pass the IoT hub primary connection string.
Create a CancellationToken.
Call GetFileNotificationReceiver to create a notification receiver.
Use a loop with ReceiveAsync to wait for the file upload notification.

For example:

using Microsoft.Azure.Devices;
static ServiceClient serviceClient;
static string connectionString = "{IoT hub connection string}";
serviceClient = ServiceClient.CreateFromConnectionString(connectionString);

// Define the cancellation token
CancellationTokenSource source = new CancellationTokenSource();
CancellationToken token = source.Token;

// Create a notification receiver
var notificationReceiver = serviceClient.GetFileNotificationReceiver();
Console.WriteLine("\nReceiving file upload notification from service");

// Check for file upload notifications
while (true)
{
    var fileUploadNotification = await notificationReceiver.ReceiveAsync(token);
    if (fileUploadNotification == null) continue;
    Console.ForegroundColor = ConsoleColor.Yellow;
    Console.WriteLine("Received file upload notification: {0}", 
        string.Join(", ", fileUploadNotification.BlobName));
    Console.ResetColor();
    await notificationReceiver.CompleteAsync(fileUploadNotification);
}

Overview

This how-to contains two sections:

Upload a file from a device application
Receive file upload notification in a backend application

Upload a file from a device application

This section describes how to upload a file from a device to an IoT hub using the DeviceClient class from the Azure IoT SDK for Java.

Follow this procedure to upload a file from a device to IoT hub:

Connect to the device
Get a SAS URI from IoT hub
Upload the file to Azure Storage
Send file upload status notification to IoT hub

Connection protocol

File upload operations always use HTTPS, but DeviceClient can define the IotHubClientProtocol for other services like telemetry, device method, and device twin.

IotHubClientProtocol protocol = IotHubClientProtocol.MQTT;

Connect to the device

Instantiate the DeviceClient to connect to the device using the device primary connection string.

String connString = "{IoT hub connection string}";
DeviceClient client = new DeviceClient(connString, protocol);

Get a SAS URI from IoT hub

Call getFileUploadSasUri to obtain a FileUploadSasUriResponse object.

FileUploadSasUriResponse includes these methods and return values. The return values can be passed to file upload methods.

Method	Return value
`getCorrelationId()`	Correlation ID
`getContainerName()`	Container name
`getBlobName()`	Blob name
`getBlobUri()`	Blob URI

For example:

FileUploadSasUriResponse sasUriResponse = client.getFileUploadSasUri(new FileUploadSasUriRequest(file.getName()));

System.out.println("Successfully got SAS URI from IoT hub");
System.out.println("Correlation Id: " + sasUriResponse.getCorrelationId());
System.out.println("Container name: " + sasUriResponse.getContainerName());
System.out.println("Blob name: " + sasUriResponse.getBlobName());
System.out.println("Blob Uri: " + sasUriResponse.getBlobUri());

Upload the file to Azure Storage

Pass the blob URI endpoint to BlobClientBuilder.buildclient to create the BlobClient object.

BlobClient blobClient =
    new BlobClientBuilder()
        .endpoint(sasUriResponse.getBlobUri().toString())
        .buildClient();

Call uploadFromFile to upload the file to Blob Storage.

String fullFileName = "Path of the file to upload";
blobClient.uploadFromFile(fullFileName);

Send file upload status notification to IoT hub

Send an upload status notification to IoT hub after a file upload attempt.

Create a FileUploadCompletionNotification object. Pass the correlationId and isSuccess file upload success status. Pass an isSuccess true value when file upload was successful, false when not.

FileUploadCompletionNotification must be called even when the file upload fails. IoT hub has a fixed number of SAS URI allowed to be active at any given time. Once you're done with the file upload, you should free your SAS URI so that other SAS URI can be generated. If a SAS URI isn't freed through this API, then it frees itself eventually based on how long SAS URIs are configured to live on an IoT hub.

This example passes a successful status.

FileUploadCompletionNotification completionNotification = new FileUploadCompletionNotification(sasUriResponse.getCorrelationId(), true);
client.completeFileUpload(completionNotification);

Close the client

Free the client resources.

client.closeNow();

Receive a file upload notification in a backend application

You can create a backend application to receive file upload notifications.

To create a file upload notification application:

Connect to the IoT hub service client
Check for a file upload notification

The ServiceClient class contains methods that services can use to receive file upload notifications.

Connect to the IoT hub service client

Create a IotHubServiceClientProtocol object. The connection uses the AMQPS protocol.

Call createFromConnectionString to connect to IoT hub. Pass the IoT hub primary connection string.

private static final String connectionString = "{IoT hub primary connection string}";
private static final IotHubServiceClientProtocol protocol = IotHubServiceClientProtocol.AMQPS;
ServiceClient sc = ServiceClient.createFromConnectionString(connectionString, protocol);

Check for file upload status

To check for file upload status:

Create a getFileUploadNotificationReceiver object.
Use open to connect to IoT hub.
Call receive to check for the file upload status. This method returns a fileUploadNotification object. If an upload notice is received, you can view upload status fields using fileUploadNotification methods.

For example:

FileUploadNotificationReceiver receiver = sc.getFileUploadNotificationReceiver();
receiver.open();
FileUploadNotification fileUploadNotification = receiver.receive(2000);

if (fileUploadNotification != null)
{
    System.out.println("File Upload notification received");
    System.out.println("Device Id : " + fileUploadNotification.getDeviceId());
    System.out.println("Blob Uri: " + fileUploadNotification.getBlobUri());
    System.out.println("Blob Name: " + fileUploadNotification.getBlobName());
    System.out.println("Last Updated : " + fileUploadNotification.getLastUpdatedTimeDate());
    System.out.println("Blob Size (Bytes): " + fileUploadNotification.getBlobSizeInBytes());
    System.out.println("Enqueued Time: " + fileUploadNotification.getEnqueuedTimeUtcDate());
}
else
{
    System.out.println("No file upload notification");
}

// Close the receiver object
receiver.close();

SDK file upload samples

There are two Java file upload samples.

Install packages

The azure-iot-device library must be installed before calling any related code.

pip install azure-iot-device

The azure.storage.blob package is used to perform the file upload.

pip install azure.storage.blob

Upload file from a device application

This section describes how to upload a file from a device to an IoT hub using the IoTHubDeviceClient class from the Azure IoT SDK for Python.

Follow this procedure to upload a file from a device to IoT hub:

Connect to the device
Get Blob Storage information
Upload the file to Blob Storage
Notify IoT hub of upload status

Import libraries

import os
from azure.iot.device import IoTHubDeviceClient
from azure.core.exceptions import AzureError
from azure.storage.blob import BlobClient

Connect to the device

To connect to the device:

Call create_from_connection_string to add the device primary connection string.
Call connect to connect the device client.

For example:

# Add your IoT hub primary connection string
CONNECTION_STRING = "{Device primary connection string}"
device_client = IoTHubDeviceClient.create_from_connection_string(CONNECTION_STRING)

# Connect the client
device_client.connect()

Get Blob Storage information

Call get_storage_info_for_blob to get information from an IoT hub about a linked Azure Storage account. This information includes the hostname, container name, blob name, and a SAS token. The get_storage_info_for_blob method also returns a correlation_id, which is used in the notify_blob_upload_status method. The correlation_id is IoT Hub's way of marking which Blob you're working on.

# Get the storage info for the blob
PATH_TO_FILE = "{Full path to local file}"
blob_name = os.path.basename(PATH_TO_FILE)
blob_info = device_client.get_storage_info_for_blob(blob_name)

Upload a file into Blob Storage

To upload a file into Blob Storage:

Use from_blob_url to create a BlobClient object from a blob URL.
Call upload_blob to upload the file into the Blob Storage.

This example parses the blob_info structure to create a URL that it uses to initialize an BlobClient. Then it calls upload_blob to upload the file into Blob Storage.

try:
    sas_url = "https://{}/{}/{}{}".format(
        blob_info["hostName"],
        blob_info["containerName"],
        blob_info["blobName"],
        blob_info["sasToken"]
    )

    print("\nUploading file: {} to Azure Storage as blob: {} in container {}\n".format(file_name, blob_info["blobName"], blob_info["containerName"]))

    # Upload the specified file
    with BlobClient.from_blob_url(sas_url) as blob_client:
        with open(file_name, "rb") as f:
            result = blob_client.upload_blob(f, overwrite=True)
            return (True, result)

except FileNotFoundError as ex:
    # catch file not found and add an HTTP status code to return in notification to IoT hub
    ex.status_code = 404
    return (False, ex)

except AzureError as ex:
    # catch Azure errors that might result from the upload operation
    return (False, ex)

Notify IoT hub of upload status

Use notify_blob_upload_status to notify IoT hub of the status of the Blob Storage operation. Pass the correlation_id obtained by the get_storage_info_for_blob method. The correlation_id is used by IoT hub to notify any service that might be listening for a notification regarding the status of the file upload task.

This example notifies IoT hub of a successful file upload:

device_client.notify_blob_upload_status(storage_info["correlationId"], True, 200, "OK: {}".format(PATH_TO_FILE)

Shut down the device client

Shut down the client. Once this method is called, any attempt at further client calls result in a ClientError being raised.

device_client.shutdown()

SDK file upload samples

The SDK includes two file upload samples:

Overview

This how-to contains two sections:

Upload a file from a device application
Receive file upload notification in a backend application

Upload a file from a device application

This section describes how to upload a file from a device to an IoT hub using the azure-iot-device package in the Azure IoT SDK for Node.js.

Install SDK packages

Run this command to install the azure-iot-device device SDK, the azure-iot-device-mqtt, and the @azure/storage-blob packages on your development machine:

npm install azure-iot-device azure-iot-device-mqtt @azure/storage-blob --save

The azure-iot-device package contains objects that interface with IoT devices.

Follow this procedure for uploading a file from a device to IoT hub:

Get Blob shared access signatures
Upload the file to Azure Storage
Send file upload status notification to IoT hub

Create modules

Create Client, Protocol, errors, and path modules using the installed packages.

const Client = require('azure-iot-device').Client;
const Protocol = require('azure-iot-device-mqtt').Mqtt;
const errors = require('azure-iot-common').errors;
const path = require('path');

Get a SAS URI from IoT hub

Use getBlobSharedAccessSignature to get the linked storage account SAS token from IoT hub. As described in prerequisites, the IoT hub is linked to the Blob Storage.

For example:

// make sure you set these environment variables prior to running the sample.
const localFilePath = process.env.PATH_TO_FILE;
const storageBlobName = path.basename(localFilePath);
const blobInfo = await client.getBlobSharedAccessSignature(storageBlobName);
if (!blobInfo) {
throw new errors.ArgumentError('Invalid upload parameters');
}

Upload the file to IoT hub

To upload a file from a device to IoT hub:

Create a stream pipeline
Construct the blob URL
Create a BlockBlobClient for file upload to Blob Storage
Call uploadFile to upload the file to Blob Storage
Call notifyBlobUploadStatus to notify IoT hub that the upload succeeded or failed

For example:

// Open the pipeline
const pipeline = newPipeline(new AnonymousCredential(), {
retryOptions: { maxTries: 4 },
telemetry: { value: 'HighLevelSample V1.0.0' }, // Customized telemetry string
keepAliveOptions: { enable: false }
});

// Construct the blob URL
const { hostName, containerName, blobName, sasToken } = blobInfo;
const blobUrl = `https://${hostName}/${containerName}/${blobName}${sasToken}`;

// Create the BlockBlobClient for file upload to Blob Storage
const blobClient = new BlockBlobClient(blobUrl, pipeline);

// Setup blank status notification arguments to be filled in on success/failure
let isSuccess;
let statusCode;
let statusDescription;

const uploadStatus = await blobClient.uploadFile(localFilePath);
console.log('uploadStreamToBlockBlob success');

  try {
    const uploadStatus = await blobClient.uploadFile(localFilePath);
    console.log('uploadStreamToBlockBlob success');

    // Save successful status notification arguments
    isSuccess = true;
    statusCode = uploadStatus._response.status;
    statusDescription = uploadStatus._response.bodyAsText;

    // Notify IoT hub of upload to blob status (success)
    console.log('notifyBlobUploadStatus success');
  }
  catch (err) {
    isSuccess = false;
    statusCode = err.code;
    statusDescription = err.message;

    console.log('notifyBlobUploadStatus failed');
    console.log(err);
  }

// Send file upload status notification to IoT hub
await client.notifyBlobUploadStatus(blobInfo.correlationId, isSuccess, statusCode, statusDescription);

Receive file upload notification in a backend application

You can create a backend application to check the IoT hub service client for device file upload notifications.

To create a file upload notification application:

Connect to the IoT hub service client
Check for a file upload notification

Connect to the IoT hub service client

The ServiceClient class contains methods that services can use to receive file upload notifications.

Connect to IoT hub using fromConnectionString. Pass the IoT hub primary connection string.

const Client = require('azure-iothub').Client;
const connectionString = "{IoT hub primary connection string}";
const serviceClient = Client.fromConnectionString(connectionString);

Open the connection to IoT hub.

//Open the connection to IoT hub
serviceClient.open(function (err) {
  if (err) {
    console.error('Could not connect: ' + err.message);
  } else {
    console.log('Service client connected');

Check for a file upload notification

To check for file upload notifications:

Call getFileNotificationReceiver. Supply the name of a file upload callback method that are called when notification messages are received.
Process file upload notifications in the callback method.

This example sets up a receiveFileUploadNotification notification callback receiver. The receiver interprets the file upload status information and prints a status message to the console.

//Set up the receiveFileUploadNotification notification message callback receiver
serviceClient.getFileNotificationReceiver(function receiveFileUploadNotification(err, receiver){
if (err) {
  console.error('error getting the file notification receiver: ' + err.toString());
} else {
  receiver.on('message', function (msg) {
    console.log('File upload from device:')
    console.log(msg.getData().toString('utf-8'));
    receiver.complete(msg, function (err) {
      if (err) {
        console.error('Could not finish the upload: ' + err.message);
      } else {
        console.log('Upload complete');
      }
    });
  });
}

SDK file upload sample

The SDK includes an upload to blob advanced sample.

Share via

Upload files from a device to the cloud with Azure IoT Hub

Prerequisites

Overview

Upload a file from a device application

Connect to the device

Get a SAS URI from IoT hub

Upload a file to Azure storage

Notify IoT hub of the file upload status

SDK file upload sample

Receive a file upload notification in a backend application

Overview

Upload a file from a device application

Connection protocol

Connect to the device

Get a SAS URI from IoT hub

Upload the file to Azure Storage

Send file upload status notification to IoT hub

Close the client

Receive a file upload notification in a backend application

Connect to the IoT hub service client

Check for file upload status

SDK file upload samples

Install packages

Upload file from a device application

Import libraries

Connect to the device

Get Blob Storage information

Upload a file into Blob Storage

Notify IoT hub of upload status

Shut down the device client

SDK file upload samples

Overview

Upload a file from a device application

Install SDK packages

Create modules

Get a SAS URI from IoT hub

Upload the file to IoT hub

Receive file upload notification in a backend application

Connect to the IoT hub service client

Check for a file upload notification

SDK file upload sample

Feedback

Additional resources