Upload files from a device to the cloud with Azure IoT Hub
This article demonstrates how to:
- Use file upload capabilities of IoT Hub to upload a file to Azure Blob Storage, using an Azure IoT device and service SDKs.
- Notify IoT Hub that the file was successfully uploaded and create a backend service to receive file upload notifications from IoT Hub, using the Azure IoT service SDKs.
In some scenarios, you can't easily map the data your devices send into the relatively small device-to-cloud messages that IoT Hub accepts. The file upload capabilities in IoT Hub enable you to move large or complex data to the cloud. For example:
- Videos
- Large files that contain images
- Vibration data sampled at high frequency
- Some form of preprocessed data
These files are typically batch processed in the cloud, using tools such as Azure Data Factory or the Hadoop stack. When you need to upload files from a device, you can still use the security and reliability of IoT Hub. This article shows you how.
This article is meant to complement runnable SDK samples that are referenced from within this article.
For more information, see:
Important
File upload functionality on devices that use X.509 certificate authority (CA) authentication is in public preview, and preview mode must be enabled. It is generally available on devices that use X.509 thumbprint authentication or X.509 certificate attestation with Azure Device Provisioning Service. To learn more about X.509 authentication with IoT Hub, see Supported X.509 certificates.
Prerequisites
An IoT hub. Some SDK calls require the IoT Hub primary connection string, so make a note of the connection string.
A registered device. Some SDK calls require the device primary connection string, so make a note of the connection string.
IoT Hub Service Connect permission - To receive file upload notification messages, your backend service needs the Service Connect permission. By default, every IoT Hub is created with a shared access policy named service that grants this permission. For more information, see Connect to an IoT hub.
Configure file upload in your IoT hub by linking an Azure Storage account and Azure Blob Storage container. You can configure these using the Azure portal, Azure CLI, or Azure PowerShell.
Overview
This how-to contains two sections:
- Upload a file from a device application
- Receive file upload notification in a backend application
Upload a file from a device application
This section describes how to upload a file from a device to an IoT hub using the DeviceClient class in the Azure IoT SDK for .NET.
Follow this procedure to upload a file from a device to IoT hub:
- Connect to IoT hub
- Get a SAS URI from IoT hub
- Upload the file to Azure storage
- Notify IoT hub of the file upload status
Connect to the device
Call CreateFromConnectionString to connect to the device. Pass the device primary connection string.
AMQP
is the default transport protocol.
static string connectionString = "{device primary connection string}";
deviceClient = DeviceClient.CreateFromConnectionString(connectionString);
Get a SAS URI from IoT hub
Call GetFileUploadSasUriAsync to get file upload details. The SAS URI is used in the next step to upload a file from a device to Blob Storage.
const string filePath = "TestPayload.txt";
using var fileStreamSource = new FileStream(filePath, FileMode.Open);
var fileName = Path.GetFileName(fileStreamSource.Name);
var fileUploadSasUriRequest = new FileUploadSasUriRequest
{
BlobName = fileName
};
FileUploadSasUriResponse sasUri = await _deviceClient.GetFileUploadSasUriAsync(fileUploadSasUriRequest, System.Threading.CancellationToken cancellationToken = default);
Uri uploadUri = sasUri.GetBlobUri();
Upload a file to Azure storage
To upload a file to Azure storage:
Create a blockBlobClient object, passing a file upload URI.
Use the UploadAsync method to upload a file to Blob Storage, passing the SAS URI. You can optionally add Blob upload options and cancellation token parameters.
The Azure Blob client always uses HTTPS as the protocol to upload the file to Azure Storage.
In this example, BlockBlobClient
is passed the SAS URI to create an Azure Storage block Blob client and uploads the file:
var blockBlobClient = new BlockBlobClient(uploadUri);
await blockBlobClient.UploadAsync(fileStreamSource, null, null);
Notify IoT hub of the file upload status
Use CompleteFileUploadAsync to notify IoT hub that the device client completed the upload, passing a FileUploadCompletionNotification object. The IsSuccess
flag indicates whether or not the upload was successful. After being notified, IoT hub will release resources associated with the upload (the SAS URI).
If file upload notifications are enabled, IoT hub sends a file upload notification message to backend services that are configured for file upload notification.
var successfulFileUploadCompletionNotification = new FileUploadCompletionNotification
{
// Mandatory. Must be the same value as the correlation id returned in the sas uri response
CorrelationId = sasUri.CorrelationId,
// Mandatory. Will be present when service client receives this file upload notification
IsSuccess = true,
// Optional, user defined status code. Will be present when service client receives this file upload notification
StatusCode = 200,
// Optional, user-defined status description. Will be present when service client receives this file upload notification
StatusDescription = "Success"
};
await _deviceClient.CompleteFileUploadAsync(successfulFileUploadCompletionNotification);
SDK file upload sample
The SDK includes this file upload sample.
Receive a file upload notification in a backend application
You can create a backend service to receive file upload notification messages from IoT hub.
The ServiceClient class contains methods that services can use to receive file upload notifications.
To receive file upload notification:
- Call CreateFromConnectionString to connect to IoT hub. Pass the IoT hub primary connection string.
- Create a CancellationToken.
- Call GetFileNotificationReceiver to create a notification receiver.
- Use a loop with ReceiveAsync to wait for the file upload notification.
For example:
using Microsoft.Azure.Devices;
static ServiceClient serviceClient;
static string connectionString = "{IoT hub connection string}";
serviceClient = ServiceClient.CreateFromConnectionString(connectionString);
// Define the cancellation token
CancellationTokenSource source = new CancellationTokenSource();
CancellationToken token = source.Token;
// Create a notification receiver
var notificationReceiver = serviceClient.GetFileNotificationReceiver();
Console.WriteLine("\nReceiving file upload notification from service");
// Check for file upload notifications
while (true)
{
var fileUploadNotification = await notificationReceiver.ReceiveAsync(token);
if (fileUploadNotification == null) continue;
Console.ForegroundColor = ConsoleColor.Yellow;
Console.WriteLine("Received file upload notification: {0}",
string.Join(", ", fileUploadNotification.BlobName));
Console.ResetColor();
await notificationReceiver.CompleteAsync(fileUploadNotification);
}
Overview
This how-to contains two sections:
- Upload a file from a device application
- Receive file upload notification in a backend application
Upload a file from a device application
This section describes how to upload a file from a device to an IoT hub using the DeviceClient class from the Azure IoT SDK for Java.
Follow this procedure to upload a file from a device to IoT hub:
- Connect to the device
- Get a SAS URI from IoT hub
- Upload the file to Azure Storage
- Send file upload status notification to IoT hub
Connection protocol
File upload operations always use HTTPS, but DeviceClient can define the IotHubClientProtocol for other services like telemetry, device method, and device twin.
IotHubClientProtocol protocol = IotHubClientProtocol.MQTT;
Connect to the device
Instantiate the DeviceClient
to connect to the device using the device primary connection string.
String connString = "{IoT hub connection string}";
DeviceClient client = new DeviceClient(connString, protocol);
Get a SAS URI from IoT hub
Call getFileUploadSasUri to obtain a FileUploadSasUriResponse object.
FileUploadSasUriResponse
includes these methods and return values. The return values can be passed to file upload methods.
Method | Return value |
---|---|
getCorrelationId() |
Correlation ID |
getContainerName() |
Container name |
getBlobName() |
Blob name |
getBlobUri() |
Blob URI |
For example:
FileUploadSasUriResponse sasUriResponse = client.getFileUploadSasUri(new FileUploadSasUriRequest(file.getName()));
System.out.println("Successfully got SAS URI from IoT hub");
System.out.println("Correlation Id: " + sasUriResponse.getCorrelationId());
System.out.println("Container name: " + sasUriResponse.getContainerName());
System.out.println("Blob name: " + sasUriResponse.getBlobName());
System.out.println("Blob Uri: " + sasUriResponse.getBlobUri());
Upload the file to Azure Storage
Pass the blob URI endpoint to BlobClientBuilder.buildclient to create the BlobClient object.
BlobClient blobClient =
new BlobClientBuilder()
.endpoint(sasUriResponse.getBlobUri().toString())
.buildClient();
Call uploadFromFile to upload the file to Blob Storage.
String fullFileName = "Path of the file to upload";
blobClient.uploadFromFile(fullFileName);
Send file upload status notification to IoT hub
Send an upload status notification to IoT hub after a file upload attempt.
Create a FileUploadCompletionNotification object. Pass the correlationId
and isSuccess
file upload success status. Pass an isSuccess
true
value when file upload was successful, false
when not.
FileUploadCompletionNotification
must be called even when the file upload fails. IoT hub has a fixed number of SAS URI allowed to be active at any given time. Once you're done with the file upload, you should free your SAS URI so that other SAS URI can be generated. If a SAS URI isn't freed through this API, then it frees itself eventually based on how long SAS URIs are configured to live on an IoT hub.
This example passes a successful status.
FileUploadCompletionNotification completionNotification = new FileUploadCompletionNotification(sasUriResponse.getCorrelationId(), true);
client.completeFileUpload(completionNotification);
Close the client
Free the client
resources.
client.closeNow();
Receive a file upload notification in a backend application
You can create a backend application to receive file upload notifications.
To create a file upload notification application:
- Connect to the IoT hub service client
- Check for a file upload notification
The ServiceClient class contains methods that services can use to receive file upload notifications.
Connect to the IoT hub service client
Create a IotHubServiceClientProtocol
object. The connection uses the AMQPS
protocol.
Call createFromConnectionString
to connect to IoT hub. Pass the IoT hub primary connection string.
private static final String connectionString = "{IoT hub primary connection string}";
private static final IotHubServiceClientProtocol protocol = IotHubServiceClientProtocol.AMQPS;
ServiceClient sc = ServiceClient.createFromConnectionString(connectionString, protocol);
Check for file upload status
To check for file upload status:
- Create a getFileUploadNotificationReceiver object.
- Use open to connect to IoT hub.
- Call receive to check for the file upload status. This method returns a fileUploadNotification object. If an upload notice is received, you can view upload status fields using fileUploadNotification methods.
For example:
FileUploadNotificationReceiver receiver = sc.getFileUploadNotificationReceiver();
receiver.open();
FileUploadNotification fileUploadNotification = receiver.receive(2000);
if (fileUploadNotification != null)
{
System.out.println("File Upload notification received");
System.out.println("Device Id : " + fileUploadNotification.getDeviceId());
System.out.println("Blob Uri: " + fileUploadNotification.getBlobUri());
System.out.println("Blob Name: " + fileUploadNotification.getBlobName());
System.out.println("Last Updated : " + fileUploadNotification.getLastUpdatedTimeDate());
System.out.println("Blob Size (Bytes): " + fileUploadNotification.getBlobSizeInBytes());
System.out.println("Enqueued Time: " + fileUploadNotification.getEnqueuedTimeUtcDate());
}
else
{
System.out.println("No file upload notification");
}
// Close the receiver object
receiver.close();
SDK file upload samples
There are two Java file upload samples.
Install packages
The azure-iot-device library must be installed before calling any related code.
pip install azure-iot-device
The azure.storage.blob package is used to perform the file upload.
pip install azure.storage.blob
Upload file from a device application
This section describes how to upload a file from a device to an IoT hub using the IoTHubDeviceClient class from the Azure IoT SDK for Python.
Follow this procedure to upload a file from a device to IoT hub:
- Connect to the device
- Get Blob Storage information
- Upload the file to Blob Storage
- Notify IoT hub of upload status
Import libraries
import os
from azure.iot.device import IoTHubDeviceClient
from azure.core.exceptions import AzureError
from azure.storage.blob import BlobClient
Connect to the device
To connect to the device:
Call create_from_connection_string to add the device primary connection string.
Call connect to connect the device client.
For example:
# Add your IoT hub primary connection string
CONNECTION_STRING = "{Device primary connection string}"
device_client = IoTHubDeviceClient.create_from_connection_string(CONNECTION_STRING)
# Connect the client
device_client.connect()
Get Blob Storage information
Call get_storage_info_for_blob to get information from an IoT hub about a linked Azure Storage account. This information includes the hostname, container name, blob name, and a SAS token. The get_storage_info_for_blob
method also returns a correlation_id
, which is used in the notify_blob_upload_status
method. The correlation_id
is IoT Hub's way of marking which Blob you're working on.
# Get the storage info for the blob
PATH_TO_FILE = "{Full path to local file}"
blob_name = os.path.basename(PATH_TO_FILE)
blob_info = device_client.get_storage_info_for_blob(blob_name)
Upload a file into Blob Storage
To upload a file into Blob Storage:
- Use from_blob_url to create a BlobClient object from a blob URL.
- Call upload_blob to upload the file into the Blob Storage.
This example parses the blob_info
structure to create a URL that it uses to initialize an BlobClient. Then it calls upload_blob
to upload the file into Blob Storage.
try:
sas_url = "https://{}/{}/{}{}".format(
blob_info["hostName"],
blob_info["containerName"],
blob_info["blobName"],
blob_info["sasToken"]
)
print("\nUploading file: {} to Azure Storage as blob: {} in container {}\n".format(file_name, blob_info["blobName"], blob_info["containerName"]))
# Upload the specified file
with BlobClient.from_blob_url(sas_url) as blob_client:
with open(file_name, "rb") as f:
result = blob_client.upload_blob(f, overwrite=True)
return (True, result)
except FileNotFoundError as ex:
# catch file not found and add an HTTP status code to return in notification to IoT hub
ex.status_code = 404
return (False, ex)
except AzureError as ex:
# catch Azure errors that might result from the upload operation
return (False, ex)
Notify IoT hub of upload status
Use notify_blob_upload_status to notify IoT hub of the status of the Blob Storage operation. Pass the correlation_id
obtained by the get_storage_info_for_blob
method. The correlation_id
is used by IoT hub to notify any service that might be listening for a notification regarding the status of the file upload task.
This example notifies IoT hub of a successful file upload:
device_client.notify_blob_upload_status(storage_info["correlationId"], True, 200, "OK: {}".format(PATH_TO_FILE)
Shut down the device client
Shut down the client. Once this method is called, any attempt at further client calls result in a ClientError being raised.
device_client.shutdown()
SDK file upload samples
The SDK includes two file upload samples:
Overview
This how-to contains two sections:
- Upload a file from a device application
- Receive file upload notification in a backend application
Upload a file from a device application
This section describes how to upload a file from a device to an IoT hub using the azure-iot-device package in the Azure IoT SDK for Node.js.
Install SDK packages
Run this command to install the azure-iot-device device SDK, the azure-iot-device-mqtt, and the @azure/storage-blob packages on your development machine:
npm install azure-iot-device azure-iot-device-mqtt @azure/storage-blob --save
The azure-iot-device package contains objects that interface with IoT devices.
Follow this procedure for uploading a file from a device to IoT hub:
- Get Blob shared access signatures
- Upload the file to Azure Storage
- Send file upload status notification to IoT hub
Create modules
Create Client, Protocol, errors, and path modules using the installed packages.
const Client = require('azure-iot-device').Client;
const Protocol = require('azure-iot-device-mqtt').Mqtt;
const errors = require('azure-iot-common').errors;
const path = require('path');
Get a SAS URI from IoT hub
Use getBlobSharedAccessSignature to get the linked storage account SAS token from IoT hub. As described in prerequisites, the IoT hub is linked to the Blob Storage.
For example:
// make sure you set these environment variables prior to running the sample.
const localFilePath = process.env.PATH_TO_FILE;
const storageBlobName = path.basename(localFilePath);
const blobInfo = await client.getBlobSharedAccessSignature(storageBlobName);
if (!blobInfo) {
throw new errors.ArgumentError('Invalid upload parameters');
}
Upload the file to IoT hub
To upload a file from a device to IoT hub:
- Create a stream pipeline
- Construct the blob URL
- Create a BlockBlobClient for file upload to Blob Storage
- Call uploadFile to upload the file to Blob Storage
- Call notifyBlobUploadStatus to notify IoT hub that the upload succeeded or failed
For example:
// Open the pipeline
const pipeline = newPipeline(new AnonymousCredential(), {
retryOptions: { maxTries: 4 },
telemetry: { value: 'HighLevelSample V1.0.0' }, // Customized telemetry string
keepAliveOptions: { enable: false }
});
// Construct the blob URL
const { hostName, containerName, blobName, sasToken } = blobInfo;
const blobUrl = `https://${hostName}/${containerName}/${blobName}${sasToken}`;
// Create the BlockBlobClient for file upload to Blob Storage
const blobClient = new BlockBlobClient(blobUrl, pipeline);
// Setup blank status notification arguments to be filled in on success/failure
let isSuccess;
let statusCode;
let statusDescription;
const uploadStatus = await blobClient.uploadFile(localFilePath);
console.log('uploadStreamToBlockBlob success');
try {
const uploadStatus = await blobClient.uploadFile(localFilePath);
console.log('uploadStreamToBlockBlob success');
// Save successful status notification arguments
isSuccess = true;
statusCode = uploadStatus._response.status;
statusDescription = uploadStatus._response.bodyAsText;
// Notify IoT hub of upload to blob status (success)
console.log('notifyBlobUploadStatus success');
}
catch (err) {
isSuccess = false;
statusCode = err.code;
statusDescription = err.message;
console.log('notifyBlobUploadStatus failed');
console.log(err);
}
// Send file upload status notification to IoT hub
await client.notifyBlobUploadStatus(blobInfo.correlationId, isSuccess, statusCode, statusDescription);
Receive file upload notification in a backend application
You can create a backend application to check the IoT hub service client for device file upload notifications.
To create a file upload notification application:
- Connect to the IoT hub service client
- Check for a file upload notification
Connect to the IoT hub service client
The ServiceClient class contains methods that services can use to receive file upload notifications.
Connect to IoT hub using fromConnectionString. Pass the IoT hub primary connection string.
const Client = require('azure-iothub').Client;
const connectionString = "{IoT hub primary connection string}";
const serviceClient = Client.fromConnectionString(connectionString);
Open the connection to IoT hub.
//Open the connection to IoT hub
serviceClient.open(function (err) {
if (err) {
console.error('Could not connect: ' + err.message);
} else {
console.log('Service client connected');
Check for a file upload notification
To check for file upload notifications:
- Call getFileNotificationReceiver. Supply the name of a file upload callback method that are called when notification messages are received.
- Process file upload notifications in the callback method.
This example sets up a receiveFileUploadNotification
notification callback receiver. The receiver interprets the file upload status information and prints a status message to the console.
//Set up the receiveFileUploadNotification notification message callback receiver
serviceClient.getFileNotificationReceiver(function receiveFileUploadNotification(err, receiver){
if (err) {
console.error('error getting the file notification receiver: ' + err.toString());
} else {
receiver.on('message', function (msg) {
console.log('File upload from device:')
console.log(msg.getData().toString('utf-8'));
receiver.complete(msg, function (err) {
if (err) {
console.error('Could not finish the upload: ' + err.message);
} else {
console.log('Upload complete');
}
});
});
}
SDK file upload sample
The SDK includes an upload to blob advanced sample.
Commentaires
https://aka.ms/ContentUserFeedback.
Bientôt disponible : Tout au long de 2024, nous allons supprimer progressivement GitHub Issues comme mécanisme de commentaires pour le contenu et le remplacer par un nouveau système de commentaires. Pour plus d’informations, consultezEnvoyer et afficher des commentaires pour