Export IoT data to Blob Storage

Άρθρο
03/05/2024

This article describes how to configure data export to send data to the Blob Storage service.

Use this feature to continuously export filtered and enriched IoT data from your IoT Central application. Data export pushes changes in near real time to other parts of your cloud solution for warm-path insights, analytics, and storage.

For example, you can:

Continuously export telemetry, property changes, device connectivity, device lifecycle, device template lifecycle, and audit log data in JSON format in near real time.
Filter the data streams to export data that matches custom conditions.
Enrich the data streams with custom values and property values from the device.
Transform the data streams to modify their shape and content.

Tip

When you turn on data export, you get only the data from that moment onward. To retain more historical data, turn on data export early. To manually export data from times when data export was turned off, see How to use the IoT Central REST API to query devices.

Note

In some circumstances it could take up to 60 seconds for the messages to be exported. This time is measured from when IoT Central receives the message from the underlying IoT hub to when the message is delivered to the destination endpoint.

Prerequisites

To use data export features, you must have the Data export permission.

To learn how to manage data export by using the IoT Central REST API, see How to use the IoT Central REST API to manage data exports.

Set up a Blob Storage export destination

IoT Central exports data once per minute, with each file containing the batch of changes since the previous export. Exported data is saved in JSON format. The default paths to the exported data in your storage account are:

Telemetry: {container}/{app-id}/{partition_id}/{YYYY}/{MM}/{dd}/{hh}/{mm}/{filename}
Property changes: {container}/{app-id}/{partition_id}/{YYYY}/{MM}/{dd}/{hh}/{mm}/{filename}

To browse the exported files in the Azure portal, navigate to the file and select Edit blob.

Connection options

Blob Storage destinations let you configure the connection with a connection string or a managed identity.

Tip

If the Blob Storage destination is protected by a firewall, you must use a managed identity to connect to it.

Managed identities are more secure because:

You don't store the credentials for your resource in a connection string in your IoT Central application.
The credentials are automatically tied to the lifetime of your IoT Central application.
Managed identities automatically rotate their security keys regularly.

IoT Central currently uses system-assigned managed identities.

When you configure a managed identity, the configuration includes a scope and a role:

The scope defines where you can use the managed identity. For example, you can use an Azure resource group as the scope. In this case, both the IoT Central application and the destination must be in the same resource group.
The role defines what permissions the IoT Central application is granted in the destination service. For example, for an IoT Central application to send data to an event hub, the managed identity needs the Azure Event Hubs Data Sender role assignment.

The following video provides more information about system assigned managed identities:

Caution

To export to blob storage, don't use the Storage Account Contributor as shown in the video. Use the Storage Blob Data Contributor role instead.

This article shows how to create a managed identity using the Azure CLI. You can also use the Azure portal to create a manged identity.

If you don't have an existing Azure storage account to export to, run the following script in the Azure Cloud Shell bash environment. The script creates a resource group, Azure Storage account, and blob container. The script then enables the managed identity for your IoT Central application and assigns the role it needs to access your storage account:

# Replace the storage account name with your own unique value.
SA=yourstorageaccount$RANDOM

# Replace the IoT Central app name with the name of your
# IoT Central application.
CA=your-iot-central-app

CN=exportdata
RG=centralexportresources
LOCATION=eastus

az group create -n $RG --location $LOCATION
SAID=$(az storage account create --name $SA --resource-group $RG --location $LOCATION --sku Standard_LRS --query "id" --output tsv)
az storage container create --account-name $SA --resource-group $RG --name $CN

# This assumes your IoT Central application is in the 
# default `IOTC` resource group.
az iot central app identity assign --name $CA --resource-group IOTC --system-assigned
PI=$(az iot central app identity show --name $CA --resource-group IOTC --query "principalId" --output tsv)

az role assignment create --assignee $PI --role "Storage Blob Data Contributor" --scope $SAID

az role assignment list --assignee $PI --all -o table

echo "Endpoint URI: https://$SA.blob.core.windows.net/"
echo "Container: $CN"

You can learn more about creating new Azure Blob Storage accounts or Azure Data Lake Storage v2 storage accounts. Data export can only write data to storage accounts that support block blobs. The following table shows the known compatible storage account types:

Performance Tier	Account Type
Standard	General Purpose V2
Standard	General Purpose V1
Standard	Blob storage
Premium	Block Blob storage

To further secure your blob container and only allow access from trusted services with managed identities, see Export data to a secure destination on an Azure Virtual Network.

To create the Blob Storage destination in IoT Central on the Data export page:

Select + New destination.
Select Azure Blob Storage as the destination type.
Select System-assigned managed identity as the authorization type.
Enter the endpoint URI for your storage account and the case-sensitive container name. An endpoint URI looks like: https://contosowaste.blob.core.windows.net.
Select Save.

If you don't see data arriving in your destination service, see Troubleshoot issues with data exports from your Azure IoT Central application.

If you don't have an existing Azure storage account to export to, run the following script in the Azure Cloud Shell bash environment. The script creates a resource group, Azure Storage account, and blob container. It then prints the connection string to use when you configure the data export in IoT Central:

# Replace the storage account name with your own unique value
SA=yourstorageaccount$RANDOM
CN=exportdata
RG=centralexportresources
LOCATION=eastus

az group create -n $RG --location $LOCATION
az storage account create --name $SA --resource-group $RG --location $LOCATION --sku Standard_LRS
az storage container create --account-name $SA --resource-group $RG --name $CN

CS=$(az storage account show-connection-string --resource-group $RG --name $SA --query "connectionString" --output tsv)

echo "Storage connection string: $CS"

Performance Tier	Account Type
Standard	General Purpose V2
Standard	General Purpose V1
Standard	Blob storage
Premium	Block Blob storage

To create the Blob Storage destination in IoT Central on the Data export page:

Select + New destination.
Select Azure Blob Storage as the destination type.
Select Connection string as the authorization type.
Paste in the connection string for your Blob Storage resource, and enter the case-sensitive container name if necessary.
Select Save.

Set up a data export

Now that you have a destination to export your data to, set up data export in your IoT Central application:

Sign in to your IoT Central application.
In the left pane, select Data export.

Tip

If you don't see Data export in the left pane, then you don't have permissions to configure data export in your app. Talk to an administrator to set up data export.
Select + New export.
Enter a display name for your new export, and make sure the data export is Enabled.

Choose the type of data to export. The following table lists the supported data export types:

Data type	Description	Data format
Telemetry	Export telemetry messages from devices in near-real time. Each exported message contains the full contents of the original device message, normalized.	Telemetry message format
Property changes	Export changes to device and cloud properties in near-real time. For read-only device properties, changes to the reported values are exported. For read-write properties, both reported and desired values are exported.	Property change message format
Device connectivity	Export device connected and disconnected events.	Device connectivity message format
Device lifecycle	Export device registered, deleted, provisioned, enabled, disabled, displayNameChanged, and deviceTemplateChanged events.	Device lifecycle changes message format
Device template lifecycle	Export published device template changes including created, updated, and deleted.	Device template lifecycle changes message format
Audit logs	Logs of user-initiated updates to entities in the application. To learn more, see Use audit logs to track activity in your IoT Central application	Audit log message format

Optionally, add filters to reduce the amount of data exported. There are different types of filter available for each data export type:

Type of data	Available filters
Telemetry	Filter by device name, device ID, device template, and if the device is simulated Filter stream to only contain telemetry that meets the filter conditions Filter stream to only contain telemetry from devices with properties matching the filter conditions Filter stream to only contain telemetry that has message properties meeting the filter condition. Message properties (also known as application properties) are sent in a bag of key-value pairs on each telemetry message. To create a message property filter, enter the message property key you're looking for, and specify a condition. Only telemetry messages with properties that match the specified filter condition are exported. Learn more about application properties from IoT Hub docs
Property changes	Filter by device name, device ID, device template, and if the device is simulated Filter stream to only contain property changes that meet the filter conditions
Device connectivity	Filter by device name, device ID, device template, organizations, and if the device is simulated Filter stream to only contain changes from devices with properties matching the filter conditions
Device lifecycle	Filter by device name, device ID, device template, and if the device is provisioned, enabled, or simulated Filter stream to only contain changes from devices with properties matching the filter conditions
Device template lifecycle	Filter by device template
Audit logs	N/A

Optionally, enrich exported messages with extra key-value pair metadata. The following enrichments are available for the telemetry, property changes, device connectivity, and device lifecycle data export types:
- Custom string: Adds a custom static string to each message. Enter any key, and enter any string value.
- Property, which adds to each message:
  - Device metadata such as device name, device template name, enabled, organizations, provisioned, and simulated.
  - The current device reported property or cloud property value to each message. If the exported message is from a device that doesn't have the specified property, the exported message doesn't get the enrichment.

Configure the export destination:

Select + Destination to add a destination that you've already created or select Create a new one.
To transform your data before it's exported, select + Transform. To learn more, see Transform data inside your IoT Central application for export.
Select + Destination to add up to five destinations to a single export.
When you've finished setting up your export, select Save. After a few minutes, your data appears in your destinations.

Monitor your export

In IoT Central, the Data export page lets you check the status of your exports. You can also use Azure Monitor to see how much data you're exporting and any export errors. You can access export and device health metrics in charts in the Azure portal by using, the REST API, queries in PowerShell, or the Azure CLI. Currently, you can monitor the following data export metrics in Azure Monitor:

Number of messages incoming to export before filters are applied.
Number of messages that pass through filters.
Number of messages successfully exported to destinations.
Number of errors found.

To learn more, see Monitor application health.

Data formats

The following sections describe the formats of the exported data:

Telemetry format

Each exported message contains a normalized form of the full message the device sent in the message body. The message is in JSON format and encoded as UTF-8. Information in each message includes:

applicationId: The ID of the IoT Central application.
messageSource: The source for the message - telemetry.
deviceId: The ID of the device that sent the telemetry message.
schema: The name and version of the payload schema.
templateId: The ID of the device template assigned to the device.
enqueuedTime: The time at which IoT Central received this message.
enrichments: Any enrichments set up on the export.
module: The IoT Edge module that sent this message. This field only appears if the message came from an IoT Edge module.
component: The component that sent this message. This field only appears if the capabilities sent in the message were modeled as a component in the device template
messageProperties: Other properties that the device sent with the message. These properties are sometimes referred to as application properties. Learn more from IoT Hub docs.

For Blob Storage, messages are batched and exported once per minute.

The following example shows an exported telemetry message:


{
    "applicationId": "00001111-aaaa-2222-bbbb-3333cccc4444",
    "messageSource": "telemetry",
    "deviceId": "1vzb5ghlsg1",
    "schema": "default@v1",
    "templateId": "urn:qugj6vbw5:___qbj_27r",
    "enqueuedTime": "2020-08-05T22:26:55.455Z",
    "telemetry": {
        "Activity": "running",
        "BloodPressure": {
            "Diastolic": 7,
            "Systolic": 71
        },
        "BodyTemperature": 98.73447010562934,
        "HeartRate": 88,
        "HeartRateVariability": 17,
        "RespiratoryRate": 13
    },
    "enrichments": {
      "userSpecifiedKey": "sampleValue"
    },
    "module": "VitalsModule",
    "component": "DeviceComponent",
    "messageProperties": {
      "messageProp": "value"
    }
}

Message properties

Telemetry messages have properties for metadata as well as the telemetry payload. The previous snippet shows examples of system messages such as deviceId and enqueuedTime. To learn more about the system message properties, see System Properties of device-to-cloud IoT Hub messages.

You can add properties to telemetry messages if you need to add custom metadata to your telemetry messages. For example, you need to add a timestamp when the device creates the message.

The following code snippet shows how to add the iothub-creation-time-utc property to the message when you create it on the device:

Important

The format of this timestamp must be UTC with no timezone information. For example, 2021-04-21T11:30:16Z is valid, 2021-04-21T11:30:16-07:00 is invalid.

async function sendTelemetry(deviceClient, index) {
  console.log('Sending telemetry message %d...', index);
  const msg = new Message(
    JSON.stringify(
      deviceTemperatureSensor.updateSensor().getCurrentTemperatureObject()
    )
  );
  msg.properties.add("iothub-creation-time-utc", new Date().toISOString());
  msg.contentType = 'application/json';
  msg.contentEncoding = 'utf-8';
  await deviceClient.sendEvent(msg);
}

private static void sendTemperatureTelemetry() {
  String telemetryName = "temperature";
  String telemetryPayload = String.format("{\"%s\": %f}", telemetryName, temperature);

  Message message = new Message(telemetryPayload);
  message.setContentEncoding(StandardCharsets.UTF_8.name());
  message.setContentTypeFinal("application/json");
  message.setProperty("iothub-creation-time-utc", Instant.now().toString());

  deviceClient.sendEventAsync(message, new MessageIotHubEventCallback(), message);
  log.debug("My Telemetry: Sent - {\"{}\": {}°C} with message Id {}.", telemetryName, temperature, message.getMessageId());
  temperatureReadings.put(new Date(), temperature);
}

private async Task SendTemperatureTelemetryAsync()
{
  const string telemetryName = "temperature";

  string telemetryPayload = $"{{ \"{telemetryName}\": {_temperature} }}";
  using var message = new Message(Encoding.UTF8.GetBytes(telemetryPayload))
  {
      ContentEncoding = "utf-8",
      ContentType = "application/json",
  };
  message.Properties.Add("iothub-creation-time-utc", DateTime.UtcNow.ToString("yyyy-MM-ddTHH:mm:ssZ"));
  await _deviceClient.SendEventAsync(message);
  _logger.LogDebug($"Telemetry: Sent - {{ \"{telemetryName}\": {_temperature}°C }}.");
}

async def send_telemetry_from_thermostat(device_client, telemetry_msg):
    msg = Message(json.dumps(telemetry_msg))
    msg.custom_properties["iothub-creation-time-utc"] = datetime.now(timezone.utc).isoformat()
    msg.content_encoding = "utf-8"
    msg.content_type = "application/json"
    print("Sent message")
    await device_client.send_message(msg)

Property changes format

Each message or record represents changes to device and cloud properties. Information in the exported message includes:

applicationId: The ID of the IoT Central application.
messageSource: The source for the message - properties.
messageType: Either cloudPropertyChange, devicePropertyDesiredChange, or devicePropertyReportedChange.
deviceId: The ID of the device that sent the telemetry message.
schema: The name and version of the payload schema.
enqueuedTime: The time at which IoT Central detected this change.
templateId: The ID of the device template assigned to the device.
properties: An array of properties that changed, including the names of the properties and values that changed. The component and module information is included if the property is modeled within a component or an IoT Edge module.
enrichments: Any enrichments set up on the export.

For Blob Storage, messages are batched and exported once per minute.

The following snippet shows a property change message exported to Blob Storage:

{
    "applicationId": "11112222-bbbb-3333-cccc-4444dddd5555",
    "messageSource": "properties",
    "deviceId": "Pepjmh1Hcc",
    "enqueuedTime": "2023-03-02T10:35:39.281Z",
    "enrichments": {},
    "messageType": "devicePropertyReportedChange",
    "schema": "default@v1",
    "templateId": "dtmi:azureiot:ddzig4ascxz",
    "properties": [
        {
            "component": "device_info",
            "name": "swVersion",
            "value": "12"
        },
        {
            "component": "device_info",
            "name": "osName",
            "value": "Android"
        },
        {
            "component": "device_info",
            "name": "processorArchitecture",
            "value": "arm64-v8a"
        },
        {
            "component": "device_info",
            "name": "processorManufacturer",
            "value": "unknown"
        }
    ]
}

Device connectivity changes format

Each message or record represents a connectivity event from a single device. Information in the exported message includes:

applicationId: The ID of the IoT Central application.
messageSource: The source for the message - deviceConnectivity.
messageType: Either connected or disconnected.
deviceId: The ID of the device that was changed.
schema: The name and version of the payload schema.
templateId: The ID of the device template assigned to the device.
enqueuedTime: The time at which this change occurred in IoT Central.
enrichments: Any enrichments set up on the export.

For Blob Storage, messages are batched and exported once per minute.

The following example shows an exported device connectivity message received in Azure Blob Storage.

{
  "applicationId": "00001111-aaaa-2222-bbbb-3333cccc4444",
  "messageSource": "deviceConnectivity",
  "messageType": "connected",
  "deviceId": "1vzb5ghlsg1",
  "schema": "default@v1",
  "templateId": "urn:qugj6vbw5:___qbj_27r",
  "enqueuedTime": "2021-04-05T22:26:55.455Z",
  "enrichments": {
    "userSpecifiedKey": "sampleValue"
  }
}

Device lifecycle changes format

Each message or record represents one change to a single device. Information in the exported message includes:

applicationId: The ID of the IoT Central application.
messageSource: The source for the message - deviceLifecycle.
messageType: The type of change that occurred. One of: registered, deleted, provisioned, enabled, disabled, displayNameChanged, and deviceTemplateChanged.
deviceId: The ID of the device that was changed.
schema: The name and version of the payload schema.
templateId: The ID of the device template assigned to the device.
enqueuedTime: The time at which this change occurred in IoT Central.
enrichments: Any enrichments set up on the export.

For Blob Storage, messages are batched and exported once per minute.

The following example shows an exported device lifecycle message received in Azure Blob Storage.

{
  "applicationId": "00001111-aaaa-2222-bbbb-3333cccc4444",
  "messageSource": "deviceLifecycle",
  "messageType": "registered",
  "deviceId": "1vzb5ghlsg1",
  "schema": "default@v1",
  "templateId": "urn:qugj6vbw5:___qbj_27r",
  "enqueuedTime": "2021-01-01T22:26:55.455Z",
  "enrichments": {
    "userSpecifiedKey": "sampleValue"
  }
}

Device template lifecycle changes format

Each message or record represents one change to a single published device template. Information in the exported message includes:

applicationId: The ID of the IoT Central application.
messageSource: The source for the message - deviceTemplateLifecycle.
messageType: Either created, updated, or deleted.
schema: The name and version of the payload schema.
templateId: The ID of the device template assigned to the device.
enqueuedTime: The time at which this change occurred in IoT Central.
enrichments: Any enrichments set up on the export.

For Blob Storage, messages are batched and exported once per minute.

The following example shows an exported device lifecycle message received in Azure Blob Storage.

{
  "applicationId": "00001111-aaaa-2222-bbbb-3333cccc4444",
  "messageSource": "deviceTemplateLifecycle",
  "messageType": "created",
  "schema": "default@v1",
  "templateId": "urn:qugj6vbw5:___qbj_27r",
  "enqueuedTime": "2021-01-01T22:26:55.455Z",
  "enrichments": {
    "userSpecifiedKey": "sampleValue"
  }
}

Audit log format

Each audit log message represents a user-initiated change to an auditable entity inside the IoT Central application. Information in the exported message includes:

actor: Information about the user who modified the entity.
applicationId: The ID of the IoT Central application.
messageSource: The source for the message - audit.
messageType: The type of change that occurred. One of: updated, created, deleted.
updated: Only present if messageType is updated. Provides more detail about the update.
resource: Details of the modified entity.
schema: The name and version of the payload schema.
deviceId: The ID of the device that was changed.
enqueuedTime: The time at which this change occurred in IoT Central.
enrichments: Any enrichments set up on the export.

The following example shows an exported audit log message received in Azure Blob Storage:

{
  "actor": {
    "id": "test-audit",
    "type": "apiToken"
    },
  "applicationId": "22223333-cccc-4444-dddd-5555eeee6666",
  "enqueuedTime": "2022-07-25T21:54:40.000Z",
  "enrichments": {},
  "messageSource": "audit",
  "messageType": "created",
  "resource": {
    "displayName": "Sensor 1",
    "id": "sensor",
    "type": "device"    
  },
  "schema": "default@v1"
}

Next steps

Now that you know how to export to Blob Storage, a suggested next step is to learn Export to Service Bus.

Κοινή χρήση μέσω

Export IoT data to Blob Storage

Prerequisites

Set up a Blob Storage export destination

Connection options

Create an Azure Blob Storage destination

Set up a data export

Monitor your export

Data formats

Telemetry format

Message properties

Property changes format

Device connectivity changes format

Device lifecycle changes format

Device template lifecycle changes format

Audit log format

Next steps

Σχόλια

Πρόσθετοι πόροι