Upload files from a device to the cloud with Azure IoT Hub
Article
This article demonstrates how to:
Use file upload capabilities of IoT Hub to upload a file to Azure Blob Storage, using an Azure IoT device and service SDKs.
Notify IoT Hub that the file was successfully uploaded and create a backend service to receive file upload notifications from IoT Hub, using the Azure IoT service SDKs.
In some scenarios, you can't easily map the data your devices send into the relatively small device-to-cloud messages that IoT Hub accepts. The file upload capabilities in IoT Hub enable you to move large or complex data to the cloud. For example:
Videos
Large files that contain images
Vibration data sampled at high frequency
Some form of preprocessed data
These files are typically batch processed in the cloud, using tools such as Azure Data Factory or the Hadoop stack. When you need to upload files from a device, you can still use the security and reliability of IoT Hub. This article shows you how.
This article is meant to complement runnable SDK samples that are referenced from within this article.
File upload functionality on devices that use X.509 certificate authority (CA) authentication is in public preview, and preview mode must be enabled. It is generally available on devices that use X.509 thumbprint authentication or X.509 certificate attestation with Azure Device Provisioning Service. To learn more about X.509 authentication with IoT Hub, see Supported X.509 certificates.
Prerequisites
An IoT hub. Some SDK calls require the IoT Hub primary connection string, so make a note of the connection string.
A registered device. Some SDK calls require the device primary connection string, so make a note of the connection string.
IoT Hub Service Connect permission - To receive file upload notification messages, your backend service needs the Service Connect permission. By default, every IoT Hub is created with a shared access policy named service that grants this permission. For more information, see Connect to an IoT hub.
Configure file upload in your IoT hub by linking an Azure Storage account and Azure Blob Storage container. You can configure these using the Azure portal, Azure CLI, or Azure PowerShell.
Overview
This how-to contains two sections:
Upload a file from a device application
Receive file upload notification in a backend application
Upload a file from a device application
This section describes how to upload a file from a device to an IoT hub using the DeviceClient class in the Azure IoT SDK for .NET.
Follow this procedure to upload a file from a device to IoT hub:
Connect to IoT hub
Get a SAS URI from IoT hub
Upload the file to Azure storage
Notify IoT hub of the file upload status
Connect a device to IoT Hub
A device app can authenticate with IoT Hub using the following methods:
X.509 certificate
Shared access key
Authenticate using an X.509 certificate
To connect a device to IoT Hub using an X.509 certificate:
Use DeviceAuthenticationWithX509Certificate to create an object that contains device and certificate information. DeviceAuthenticationWithX509Certificate is passed as the second parameter to DeviceClient.Create (step 2).
Use DeviceClient.Create to connect the device to IoT Hub using an X.509 certificate.
In this example, device and certificate information is populated in the authDeviceAuthenticationWithX509Certificate object that is passed to DeviceClient.Create.
This example shows certificate input parameter values as local variables for clarity. In a production system, store sensitive input parameters in environment variables or another more secure storage location. For example, use Environment.GetEnvironmentVariable("HOSTNAME") to read the host name environment variable.
RootCertPath = "~/certificates/certs/sensor-thl-001-device.cert.pem";
Intermediate1CertPath = "~/certificates/certs/sensor-thl-001-device.intermediate1.cert.pem";
Intermediate2CertPath = "~/certificates/certs/sensor-thl-001-device.intermediate2.cert.pem";
DevicePfxPath = "~/certificates/certs/sensor-thl-001-device.cert.pfx";
DevicePfxPassword = "1234";
DeviceName = "MyDevice";
HostName = "xxxxx.azure-devices.net";
var chainCerts = new X509Certificate2Collection();
chainCerts.Add(new X509Certificate2(RootCertPath));
chainCerts.Add(new X509Certificate2(Intermediate1CertPath));
chainCerts.Add(new X509Certificate2(Intermediate2CertPath));
using var deviceCert = new X509Certificate2(DevicePfxPath, DevicePfxPassword);
using var auth = new DeviceAuthenticationWithX509Certificate(DeviceName, deviceCert, chainCerts);
using var deviceClient = DeviceClient.Create(
HostName,
auth,
TransportType.Amqp);
For more information about certificate authentication, see:
Use the UploadAsync method to upload a file to Blob Storage, passing the SAS URI. You can optionally add Blob upload options and cancellation token parameters.
The Azure Blob client always uses HTTPS as the protocol to upload the file to Azure Storage.
In this example, BlockBlobClient is passed the SAS URI to create an Azure Storage block Blob client and uploads the file:
var blockBlobClient = new BlockBlobClient(uploadUri);
await blockBlobClient.UploadAsync(fileStreamSource, null, null);
Notify IoT hub of the file upload status
Use CompleteFileUploadAsync to notify IoT hub that the device client completed the upload, passing a FileUploadCompletionNotification object. The IsSuccess flag indicates whether or not the upload was successful. After being notified, IoT hub will release resources associated with the upload (the SAS URI).
If file upload notifications are enabled, IoT hub sends a file upload notification message to backend services that are configured for file upload notification.
var successfulFileUploadCompletionNotification = new FileUploadCompletionNotification
{
// Mandatory. Must be the same value as the correlation id returned in the sas uri response
CorrelationId = sasUri.CorrelationId,
// Mandatory. Will be present when service client receives this file upload notification
IsSuccess = true,
// Optional, user defined status code. Will be present when service client receives this file upload notification
StatusCode = 200,
// Optional, user-defined status description. Will be present when service client receives this file upload notification
StatusDescription = "Success"
};
await _deviceClient.CompleteFileUploadAsync(successfulFileUploadCompletionNotification);
Receive a file upload notification in a backend application
You can create a backend service to receive file upload notification messages from IoT hub.
The ServiceClient class contains methods that services can use to receive file upload notifications.
Add service NuGet Package
Backend service applications require the Microsoft.Azure.Devices NuGet package.
Connect to IoT hub
You can connect a backend service to IoT Hub using the following methods:
Shared access policy
Microsoft Entra
Important
This article includes steps to connect to a service using a shared access signature. This authentication method is convenient for testing and evaluation, but authenticating to a service with Microsoft Entra ID or managed identities is a more secure approach. To learn more, see Security best practices > Cloud security.
Connect using a shared access policy
Connect a backend application to a device using CreateFromConnectionString. Your application needs service connect permission. Supply this shared access policy connection string as a parameter to fromConnectionString. For more information about shared access policies, see Control access to IoT Hub with shared access signatures.
A backend app that uses Microsoft Entra must successfully authenticate and obtain a security token credential before connecting to IoT Hub. This token is passed to a IoT Hub connection method. For general information about setting up and using Microsoft Entra for IoT Hub, see Control access to IoT Hub by using Microsoft Entra ID.
Configure Microsoft Entra app
You must set up a Microsoft Entra app that is configured for your preferred authentication credential. The app contains parameters such as client secret that are used by the backend application to authenticate. The available app authentication configurations are:
The easiest way to use Microsoft Entra to authenticate a backend application is to use DefaultAzureCredential, but it's recommended to use a different method in a production environment including a specific TokenCredential or pared-down ChainedTokenCredential. For simplicity, this section describes authentication using DefaultAzureCredential and Client secret. For more information about the pros and cons of using DefaultAzureCredential, see Usage guidance for DefaultAzureCredential.
DefaultAzureCredential supports different authentication mechanisms and determines the appropriate credential type based on the environment it's executing in. It attempts to use multiple credential types in an order until it finds a working credential.
Microsoft Entra requires these NuGet packages and corresponding using statements:
Azure.Core
Azure.Identity
using Azure.Core;
using Azure.Identity;
In this example, Microsoft Entra app registration client secret, client ID, and tenant ID are added to environment variables. These environment variables are used by DefaultAzureCredential to authenticate the application. The result of a successful Microsoft Entra authentication is a security token credential that is passed to an IoT Hub connection method.
Add the SSLContext information to a ClientOptions object.
Call DeviceClient using the ClientOptions information to create the device-to-IoT Hub connection.
This example shows certificate input parameter values as local variables for clarity. In a production system, store sensitive input parameters in environment variables or another more secure storage location. For example, use Environment.GetEnvironmentVariable("PUBLICKEY") to read a public key certificate string environment variable.
File upload operations always use HTTPS, but DeviceClient can define the IotHubClientProtocol for other services like telemetry, device method, and device twin.
String fullFileName = "Path of the file to upload";
blobClient.uploadFromFile(fullFileName);
Send file upload status notification to IoT hub
Send an upload status notification to IoT hub after a file upload attempt.
Create a FileUploadCompletionNotification object. Pass the correlationId and isSuccess file upload success status. Pass an isSuccesstrue value when file upload was successful, false when not.
FileUploadCompletionNotification must be called even when the file upload fails. IoT hub has a fixed number of SAS URI allowed to be active at any given time. Once you're done with the file upload, you should free your SAS URI so that other SAS URI can be generated. If a SAS URI isn't freed through this API, then it frees itself eventually based on how long SAS URIs are configured to live on an IoT hub.
This example passes a successful status.
FileUploadCompletionNotification completionNotification = new FileUploadCompletionNotification(sasUriResponse.getCorrelationId(), true);
client.completeFileUpload(completionNotification);
Close the client
Free the client resources.
client.closeNow();
Create a backend application
This section describes how to receive a file upload notification in a backend application.
The ServiceClient class contains methods that services can use to receive file upload notifications.
Add import statements
Add these import statements to use the Azure IoT Java SDK and exception handler.
You can connect a backend service to IoT Hub using the following methods:
Shared access policy
Microsoft Entra
Important
This article includes steps to connect to a service using a shared access signature. This authentication method is convenient for testing and evaluation, but authenticating to a service with Microsoft Entra ID or managed identities is a more secure approach. To learn more, see Security best practices > Cloud security.
Connect using a shared access policy
Define the connection protocol
Use IotHubServiceClientProtocol to define the application-layer protocol used by the service client to communicate with an IoT Hub.
IotHubServiceClientProtocol only accepts the AMQPS or AMQPS_WS enum.
private static final IotHubServiceClientProtocol protocol =
IotHubServiceClientProtocol.AMQPS;
Create the ServiceClient object
Create the ServiceClient object, supplying the Iot Hub connection string and protocol.
To upload a file on a device to IoT Hub, your service needs the service connect permission. By default, every IoT Hub is created with a shared access policy named service that grants this permission.
Open the connection between application and IoT Hub
Open the AMQP sender connection. This method creates the connection between the application and IoT Hub.
serviceClient.open();
Connect using Microsoft Entra
A backend app that uses Microsoft Entra must successfully authenticate and obtain a security token credential before connecting to IoT Hub. This token is passed to a IoT Hub connection method. For general information about setting up and using Microsoft Entra for IoT Hub, see Control access to IoT Hub by using Microsoft Entra ID.
For simplicity, this section focuses on describing authentication using client secret.
Configure Microsoft Entra app
You must set up a Microsoft Entra app that is configured for your preferred authentication credential. The app contains parameters such as client secret that are used by the backend application to authenticate. The available app authentication configurations are:
The easiest way to use Microsoft Entra to authenticate a backend application is to use DefaultAzureCredential, but it's recommended to use a different method in a production environment including a specific TokenCredential or pared-down ChainedTokenCredential.
For more information about the pros and cons of using DefaultAzureCredential, see
Credential chains in the Azure Identity client library for Java.
DefaultAzureCredential supports different authentication mechanisms and determines the appropriate credential type based on the environment it's executing in. It attempts to use multiple credential types in an order until it finds a working credential.
You can authenticate Microsoft Entra app credentials using DefaultAzureCredentialBuilder. Save connection parameters such as client secret tenantID, clientID, and client secret values as environmental variables. Once the TokenCredential is created, pass it to ServiceClient or other builder as the 'credential' parameter.
In this example, DefaultAzureCredentialBuilder attempts to authenticate a connection from the list described in DefaultAzureCredential. The result of a successful Microsoft Entra authentication is a security token credential that is passed to a constructor such as ServiceClient.
TokenCredential defaultAzureCredential = new DefaultAzureCredentialBuilder().build();
Authenticate using ClientSecretCredentialBuilder
You can use ClientSecretCredentialBuilder to create a credential using client secret information. If successful, this method returns a TokenCredential that can be passed to ServiceClient or other builder as the 'credential' parameter.
In this example, Microsoft Entra app registration client secret, client ID, and tenant ID values have been added to environment variables. These environment variables are used by ClientSecretCredentialBuilder to build the credential.
Call receive to check for the file upload status. This method returns a fileUploadNotification object. If an upload notice is received, you can view upload status fields using fileUploadNotification methods.
This example shows certificate input parameter values as local variables for clarity. In a production system, store sensitive input parameters in environment variables or another more secure storage location. For example, use os.getenv("HOSTNAME") to read the host name environment variable.
# The Azure IoT hub name
hostname = "xxxxx.azure-devices.net"
# The device that has been created on the portal using X509 CA signing or self-signing capabilities
device_id = "MyDevice"
# The X.509 certificate file name
cert_file = "~/certificates/certs/sensor-thl-001-device.cert.pfx"
key_file = "~/certificates/certs/sensor-thl-001-device.cert.key"
# The optional certificate pass phrase
pass_phrase = "1234"
x509 = X509(
cert_file,
key_file,
pass_phrase,
)
# The client object is used to interact with your Azure IoT hub.
device_client = IoTHubDeviceClient.create_from_x509_certificate(
hostname=hostname, device_id=device_id, x509=x509
)
# Connect to IoT Hub
await device_client.connect()
For more information about certificate authentication, see:
Call get_storage_info_for_blob to get information from an IoT hub about a linked Azure Storage account. This information includes the hostname, container name, blob name, and a SAS token. The get_storage_info_for_blob method also returns a correlation_id, which is used in the notify_blob_upload_status method. The correlation_id is IoT Hub's way of marking which Blob you're working on.
# Get the storage info for the blob
PATH_TO_FILE = "{Full path to local file}"
blob_name = os.path.basename(PATH_TO_FILE)
blob_info = device_client.get_storage_info_for_blob(blob_name)
Call upload_blob to upload the file into the Blob Storage.
This example parses the blob_info structure to create a URL that it uses to initialize an BlobClient. Then it calls upload_blob to upload the file into Blob Storage.
try:
sas_url = "https://{}/{}/{}{}".format(
blob_info["hostName"],
blob_info["containerName"],
blob_info["blobName"],
blob_info["sasToken"]
)
print("\nUploading file: {} to Azure Storage as blob: {} in container {}\n".format(file_name, blob_info["blobName"], blob_info["containerName"]))
# Upload the specified file
with BlobClient.from_blob_url(sas_url) as blob_client:
with open(file_name, "rb") as f:
result = blob_client.upload_blob(f, overwrite=True)
return (True, result)
except FileNotFoundError as ex:
# catch file not found and add an HTTP status code to return in notification to IoT hub
ex.status_code = 404
return (False, ex)
except AzureError as ex:
# catch Azure errors that might result from the upload operation
return (False, ex)
Notify IoT hub of upload status
Use notify_blob_upload_status to notify IoT hub of the status of the Blob Storage operation. Pass the correlation_id obtained by the get_storage_info_for_blob method. The correlation_id is used by IoT hub to notify any service that might be listening for a notification regarding the status of the file upload task.
This example notifies IoT hub of a successful file upload:
This article describes how to use the Azure IoT SDK for Node.js to create a device app to upload a file and backend service application receive file upload notification.
Create a device application
This section describes how to upload a file from a device to an IoT hub using the azure-iot-device package in the Azure IoT SDK for Node.js.
Install SDK packages
Run this command to install the azure-iot-device device SDK, the azure-iot-device-mqtt, and the @azure/storage-blob packages on your development machine:
A device app can authenticate with IoT Hub using the following methods:
X.509 certificate
Shared access key
Authenticate using an X.509 certificate
The X.509 certificate is attached to the device-to-IoT Hub connection transport.
To configure a device-to-IoT Hub connection using an X.509 certificate:
Call fromConnectionString to add the device or identity module connection string, and transport type to the Client object. Add x509=true to the connection string to indicate that a certificate is added to DeviceClientOptions. For example:
Configure a JSON variable with certificate details and pass it to DeviceClientOptions.
Call setOptions to add an X.509 certificate and key (and optionally, passphrase) to the client transport.
Call open to open the connection from the device to IoT Hub.
This example shows certificate configuration information within a JSON variable. The certification configuration clientOptions are passed to setOptions, and the connection is opened using open.
const Client = require('azure-iot-device').Client;
const Protocol = require('azure-iot-device-mqtt').Mqtt;
// Connection string illustrated for demonstration only. Never hard-code the connection string in production. Instead use an environmental variable or other secure storage.
const connectionString = `HostName=xxxxx.azure-devices.net;DeviceId=Device-1;SharedAccessKey=xxxxxxxxxxxxx;x509=true`
const client = Client.fromConnectionString(connectionString, Protocol);
var clientOptions = {
cert: myX509Certificate,
key: myX509Key,
passphrase: passphrase,
http: {
receivePolicy: {
interval: 10
}
}
}
client.setOptions(clientOptions);
client.open(connectCallback);
For more information about certificate authentication, see:
// make sure you set these environment variables prior to running the sample.
const localFilePath = process.env.PATH_TO_FILE;
const storageBlobName = path.basename(localFilePath);
const blobInfo = await client.getBlobSharedAccessSignature(storageBlobName);
if (!blobInfo) {
throw new errors.ArgumentError('Invalid upload parameters');
}
This section describes how to receive file upload notifications in a backend application.
The ServiceClient class contains methods that services can use to receive file upload notifications.
Install service SDK package
Run this command to install azure-iothub on your development machine:
npm install azure-iothub --save
Connect to IoT hub
You can connect a backend service to IoT Hub using the following methods:
Shared access policy
Microsoft Entra
Important
This article includes steps to connect to a service using a shared access signature. This authentication method is convenient for testing and evaluation, but authenticating to a service with Microsoft Entra ID or managed identities is a more secure approach. To learn more, see Security best practices > Cloud security.
To upload a file from a device, your service needs the service connect permission. By default, every IoT Hub is created with a shared access policy named service that grants this permission.
var Client = require('azure-iothub').Client;
var connectionString = '{IoT hub shared access policy connection string}';
var client = Client.fromConnectionString(connectionString);
Connect using Microsoft Entra
A backend app that uses Microsoft Entra must successfully authenticate and obtain a security token credential before connecting to IoT Hub. This token is passed to a IoT Hub connection method. For general information about setting up and using Microsoft Entra for IoT Hub, see Control access to IoT Hub by using Microsoft Entra ID.
For an overview of Node.js SDK authentication, see:
You must set up a Microsoft Entra app that is configured for your preferred authentication credential. The app contains parameters such as client secret that are used by the backend application to authenticate. The available app authentication configurations are:
The easiest way to use Microsoft Entra to authenticate a backend application is to use DefaultAzureCredential, but it's recommended to use a different method in a production environment including a specific TokenCredential or pared-down ChainedTokenCredential. For simplicity, this section describes authentication using DefaultAzureCredential and Client secret.
For more information about the pros and cons of using DefaultAzureCredential, see
Credential chains in the Azure Identity client library for JavaScript
DefaultAzureCredential supports different authentication mechanisms and determines the appropriate credential type based on the environment it's executing in. It attempts to use multiple credential types in an order until it finds a working credential.
Microsoft Entra requires this package:
npm install --save @azure/identity
In this example, Microsoft Entra app registration client secret, client ID, and tenant ID have been added to environment variables. These environment variables are used by DefaultAzureCredential to authenticate the application. The result of a successful Microsoft Entra authentication is a security token credential that is passed to an IoT Hub connection method.
import { DefaultAzureCredential } from "@azure/identity";
// Azure SDK clients accept the credential as a parameter
const credential = new DefaultAzureCredential();
The resulting credential token can then be passed to fromTokenCredential to connect to IoT Hub for any SDK client that accepts Microsoft Entra credentials:
The Azure service URL - The Azure service URL should be in the format {Your Entra domain URL}.azure-devices.net without a https:// prefix. For example, MyAzureDomain.azure-devices.net.
The Azure credential token
In this example, the Azure credential is obtained using DefaultAzureCredential. The Azure domain URL and credential are then supplied to Registry.fromTokenCredential to create the connection to IoT Hub.
const { DefaultAzureCredential } = require("@azure/identity");
let Registry = require('azure-iothub').Registry;
// Define the client secret values
clientSecretValue = 'xxxxxxxxxxxxxxx'
clientID = 'xxxxxxxxxxxxxx'
tenantID = 'xxxxxxxxxxxxx'
// Set environment variables
process.env['AZURE_CLIENT_SECRET'] = clientSecretValue;
process.env['AZURE_CLIENT_ID'] = clientID;
process.env['AZURE_TENANT_ID'] = tenantID;
// Acquire a credential object
const credential = new DefaultAzureCredential()
// Create an instance of the IoTHub registry
hostName = 'MyAzureDomain.azure-devices.net';
let registry = Registry.fromTokenCredential(hostName,credential);
Create a file upload notification callback receiver
To create a file upload notification callback receiver:
Call getFileNotificationReceiver. Supply the name of a file upload callback method that is called when notification messages are received.
Process file upload notifications in the callback method.
This example sets up a receiveFileUploadNotification notification callback receiver. The receiver interprets the file upload status information and prints a status message to the console.
//Set up the receiveFileUploadNotification notification message callback receiver
serviceClient.getFileNotificationReceiver(function receiveFileUploadNotification(err, receiver){
if (err) {
console.error('error getting the file notification receiver: ' + err.toString());
} else {
receiver.on('message', function (msg) {
console.log('File upload from device:')
console.log(msg.getData().toString('utf-8'));
receiver.complete(msg, function (err) {
if (err) {
console.error('Could not finish the upload: ' + err.message);
} else {
console.log('Upload complete');
}
});
});
}
Start here and learn how you can get the full power of Azure with your Java apps - use idiomatic libraries to connect and interact with your preferred cloud services, including Azure SQL and NoSQL databases, messaging and eventing systems, Redis cache, storage and directory services. As always, use tools and frameworks that you know and love – Spring, Tomcat, WildFly, JBoss, WebLogic, WebSphere, Maven, Gradle, IntelliJ, Eclipse, Jenkins, Terraform and more.