Export IoT data to cloud destinations using data export (legacy)
The legacy data export (classic) feature is now deprecated and you should plan to migrate to the new data export feature. The legacy data export lacks important capabilities such as the availability of different data types, filtering, and message transformation. See the following table for a comparison of legacy data export with new data export:
| Capability | Legacy data export (classic) | New data export |
|---|---|---|
| Available data types | Telemetry, devices, device templates | Telemetry, property changes, device connectivity changes, device lifecycle changes, device template lifecycle changes |
| Filtering | None | Depends on the data type exported. For telemetry, filtering by telemetry, message properties, property values |
| Enrichments | None | Enrich with a custom string or a property value on the device |
| Transforms | None | Transform the export message to your desired shape |
| Destinations | Azure Event Hubs, Azure Service Bus queues and topics, Azure Blob Storage | Same as for legacy data export plus Azure Data Explorer and webhooks |
| Notable limits | Five exports per app, one destination per export | 10 export-destination connections per app |
Migration considerations
To migrate a legacy data export (classic) to new data export, you should:
- Use a test IoT Central application and create a new data export with the same data type and destination. You can optionally use the enrichments and data transformation functionality to make your export message shape similar to the message shape from your legacy data export.
- When you've tested your new data export and are ready to go to production, ensure any workflows or dependencies on your active legacy data exports are safely removed.
- Create your new data exports in your production environments and verify that the export messages are meeting your requirements. You can then add any workflows or dependencies to your new data export.
- After you've successfully migrated all your legacy data exports to new data exports, you can delete the legacy data exports.
Data type migration considerations
The default data format varies for data types between legacy data export and new data export. For more information, see data formats for new data export and data formats for legacy data export. When you migrate to the new data export, you should remove any dependencies on data format of your legacy data export. However, if you have strong dependencies or workflows tied to your legacy data exports then the following considerations can help address any migration challenges.
Telemetry: If you choose to match the legacy data export format for your telemetry in your new data export, you can use the transform functionality and build a transformation query similar to the following example:
.telemetry | map({ key: .name, value: .value }) | from_entries
Devices: If you're currently using legacy data exports with the devices data type then you can use both the property changes and device lifecycle events data types in new export to export the same data. You can achieve a comparable data structure using the following transformation query on both data types:
approved: .device.approved,
provisioned: .device.provisioned,
simulated: .device.simulated,
cloudProperties: .device.cloudProperties | map({ key: .name, value: .value }) | from_entries,
displayName: .device.name,
id: .device.id,
instanceOf: .device.templateId,
properties: .device.properties.reported | map({ key: .name, value: .value }) | from_entries
Device templates: If you're currently using legacy data exports with the device templates data type, then you can obtain the same data using the Device Templates - Get API call.
Destination migration considerations
In the new data export, you can create a destination and reuse it across different data exports. When you migrate from legacy data exports, you should create destinations in the new data exports that store information on your existing legacy data export destinations.
Note
The new data export doesn't support exporting non-valid JSON messages.
Export IoT data to cloud destinations (legacy)
Note
This article describes the legacy data export features in IoT Central
Legacy data exports (classic) are scheduled to be retired. Migrate any legacy data exports to new exports
For information about the latest data export features, see Export IoT data to cloud destinations using Blob Storage.
This article describes how to use the data export feature in Azure IoT Central. This feature lets you export your data continuously to Azure Event Hubs, Azure Service Bus, or Azure Blob storage instances. Data export uses the JSON format and can include telemetry, device information, and device template information. Use the exported data for:
- Warm-path insights and analytics. This option includes triggering custom rules in Azure Stream Analytics, triggering custom workflows in Azure Logic Apps, or passing it through Azure Functions to be transformed.
- Cold-path analytics such as training models in Azure Machine Learning or long-term trend analysis in Microsoft Power BI.
Note
When you turn on data export, you get only the data from that moment onward. Currently, data can't be retrieved for a time when data export was off. To retain more historical data, turn on data export early.
Prerequisites
You must be an administrator in your IoT Central application, or have Data export permissions.
Set up export destination
Your export destination must exist before you configure your data export.
Create Event Hubs namespace
If you don't have an existing Event Hubs namespace to export to, follow these steps:
Create a new Event Hubs namespace in the Azure portal. You can learn more in Azure Event Hubs docs.
Choose a subscription. You can export data to other subscriptions that aren't in the same subscription as your IoT Central application. You connect using a connection string in this case.
Create an event hub in your Event Hubs namespace. Go to your namespace, and select + Event Hub at the top to create an event hub instance.
Create Service Bus namespace
If you don't have an existing Service Bus namespace to export to, follow these steps:
Create a new Service Bus namespace in the Azure portal. You can learn more in Azure Service Bus docs.
Choose a subscription. You can export data to other subscriptions that aren't in the same subscription as your IoT Central application. You connect using a connection string in this case.
To create a queue or topic to export to, go to your Service Bus namespace, and select + Queue or + Topic.
When you choose Service Bus as an export destination, the queues and topics must not have Sessions or Duplicate Detection enabled. If either of those options are enabled, some messages won't arrive in your queue or topic.
Create storage account
If you don't have an existing Azure storage account to export to, follow these steps:
Create a new storage account in the Azure portal. You can learn more about creating new Azure Blob storage accounts or Azure Data Lake Storage v2 storage accounts. Data export can only write data to storage accounts that support block blobs. The following list shows the known compatible storage account types:
Performance Tier Account Type Standard General Purpose V2 Standard General Purpose V1 Standard Blob storage Premium Block Blob storage Create a container in your storage account. Go to your storage account. Under Blob Service, select Browse Blobs. Select + Container at the top to create a new container.
Set up data export
Now that you have a destination to export data to, follow these steps to set up data export.
Sign in to your IoT Central application.
In the left pane, select Data export.
Tip
If you don't see Data export in the left pane, then you don't have permissions to configure data export in your app. Talk to an administrator to set up data export.
Select the + New button. Choose one of Azure Blob Storage, Azure Event Hubs, Azure Service Bus Queue, or Azure Service Bus Topic as the destination of your export. The maximum number of exports per application is five.
Enter a name for the export. In the drop-down list box, select your namespace, or Enter a connection string.
Tip
You only see storage accounts, Event Hubs namespaces, and Service Bus namespaces in the same subscription as your IoT Central application. If you want to export to a destination outside of this subscription, choose Enter a connection string and see step 6.
Choose an event hub, queue, topic, or container from the drop-down list box.
(Optional) If you chose Enter a connection string, a new box appears for you to paste your connection string. To get the connection string for your:
- Event Hubs or Service Bus, go to the namespace in the Azure portal:
- To use a connection string for the entire namespace:
- Under Settings, select Shared Access Policies
- Create a new key or choose an existing key that has Send permissions.
- Copy either the primary or secondary connection string
- To use connection string for a specific event hub instance or Service Bus queue or topic, go to Entities > Event Hubs or Entities > Queues or Entities > Topics. Choose a specific instance, and follow the same steps above to get a connection string.
- To use a connection string for the entire namespace:
- Storage account, go to the storage account in the Azure portal:
- Only connection strings for the entire storage account are supported. Connection strings scoped to a single container aren't supported.
- Under Settings, select Access keys
- Copy either the key1 connection string or the key2 connection string
- Only connection strings for the entire storage account are supported. Connection strings scoped to a single container aren't supported.
Paste in the connection string. Type in the instance or case-sensitive container name.
- Event Hubs or Service Bus, go to the namespace in the Azure portal:
Under Data to export, choose the types of data to export by setting the type to On.
To turn on data export, make sure the Enabled toggle is On. Select Save.
After a few minutes, your data appears in your chosen destination.
Export contents and format
Exported telemetry data contains the entirety of the message your devices sent to IoT Central, not just the telemetry values themselves. Exported devices data contains changes to properties and metadata of all devices, and exported device templates contains changes to all device templates.
For Event Hubs and Service Bus, data is exported in near-realtime. The data is in the body property and is in JSON format. See below for examples.
For Blob storage, data is exported once per minute, with each file containing the batch of changes since the last exported file. Exported data is placed in three folders in JSON format. The default paths in your storage account are:
- Telemetry: {container}/{app-id}/telemetry/{YYYY}/{MM}/{dd}/{hh}/{mm}/{filename}
- Devices: {container}/{app-id}/devices/{YYYY}/{MM}/{dd}/{hh}/{mm}/{filename}
- Device templates: {container}/{app-id}/deviceTemplates/{YYYY}/{MM}/{dd}/{hh}/{mm}/{filename}
To browse the exported files in the Azure portal, navigate to the file and select the Edit blob tab.
Telemetry
For Event Hubs and Service Bus, IoT Central exports a new message quickly after it receives the message from a device. Each exported message contains the full message the device sent in the body property in JSON format.
For Blob storage, messages are batched and exported once per minute. The exported files use the same format as the message files exported by IoT Hub message routing to blob storage.
Note
For Blob storage, ensure that your devices are sending messages that have contentType: application/JSON and contentEncoding:utf-8 (or utf-16, utf-32). See the IoT Hub documentation for an example.
The device that sent the telemetry is represented by the device ID (see the following sections). To get the names of the devices, export device data and correlate each message by using the connectionDeviceId that matches the deviceId of the device message.
The following example shows a message received from an event hub or Service Bus queue or topic:
{
"temp":81.129693132351775,
"humid":59.488071477541247,
"EventProcessedUtcTime":"2020-04-07T09:41:15.2877981Z",
"PartitionId":0,
"EventEnqueuedUtcTime":"2020-04-07T09:38:32.7380000Z"
}
This message doesn't include the device ID of the sending device.
To retrieve the device ID from the message data in an Azure Stream Analytics query, use the GetMetadataPropertyValue function. For an example, see the query in Extend Azure IoT Central with custom rules using Stream Analytics, Azure Functions, and SendGrid.
To retrieve the device ID in an Azure Databricks or Apache Spark workspace, use systemProperties. For an example, see the Databricks workspace in Extend Azure IoT Central with custom analytics using Azure Databricks.
The following example shows a record exported to blob storage:
{
"EnqueuedTimeUtc":"2019-09-26T17:46:09.8870000Z",
"Properties":{
},
"SystemProperties":{
"connectionDeviceId":"<deviceid>",
"connectionAuthMethod":"{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
"connectionDeviceGenerationId":"637051167384630591",
"contentType":"application/json",
"contentEncoding":"utf-8",
"enqueuedTime":"2019-09-26T17:46:09.8870000Z"
},
"Body":{
"temp":49.91322758395974,
"humid":49.61214852573155,
"pm25":25.87332214661367
}
}
Devices
Each message or record in a snapshot represents one or more changes to a device and its device and cloud properties since the last exported message. The message includes the:
idof the device in IoT CentraldisplayNameof the device- Device template ID in
instanceOf simulatedflag, true if the device is a simulated deviceprovisionedflag, true if the device has been provisionedapprovedflag, true if the device has been approved to send data- Property values
propertiesincluding device and cloud properties values
Deleted devices aren't exported. Currently, there are no indicators in exported messages for deleted devices.
For Event Hubs and Service Bus, IoT Central sends messages containing device data to your event hub or Service Bus queue or topic in near real time.
For Blob storage, a new snapshot containing all the changes since the last one written is exported once per minute.
The following example message shows information about devices and properties data in an event hub or Service Bus queue or topic:
{
"body":{
"id": "<device Id>",
"etag": "<etag>",
"displayName": "Sensor 1",
"instanceOf": "<device template Id>",
"simulated": false,
"provisioned": true,
"approved": true,
"properties": {
"sensorComponent": {
"setTemp": "30",
"fwVersion": "2.0.1",
"status": { "first": "first", "second": "second" },
"$metadata": {
"setTemp": {
"desiredValue": "30",
"desiredVersion": 3,
"desiredTimestamp": "2020-02-01T17:15:08.9284049Z",
"ackVersion": 3
},
"fwVersion": { "ackVersion": 3 },
"status": {
"desiredValue": {
"first": "first",
"second": "second"
},
"desiredVersion": 2,
"desiredTimestamp": "2020-02-01T17:15:08.9284049Z",
"ackVersion": 2
}
},
}
},
"installDate": { "installDate": "2020-02-01" }
},
"annotations":{
"iotcentral-message-source":"devices",
"x-opt-partition-key":"<partitionKey>",
"x-opt-sequence-number":39740,
"x-opt-offset":"<offset>",
"x-opt-enqueued-time":1539274959654
},
"partitionKey":"<partitionKey>",
"sequenceNumber":39740,
"enqueuedTimeUtc":"2020-02-01T18:14:49.3820326Z",
"offset":"<offset>"
}
This snapshot is an example message that shows devices and properties data in Blob storage. Exported files contain a single line per record.
{
"id": "<device Id>",
"etag": "<etag>",
"displayName": "Sensor 1",
"instanceOf": "<device template Id>",
"simulated": false,
"provisioned": true,
"approved": true,
"properties": {
"sensorComponent": {
"setTemp": "30",
"fwVersion": "2.0.1",
"status": { "first": "first", "second": "second" },
"$metadata": {
"setTemp": {
"desiredValue": "30",
"desiredVersion": 3,
"desiredTimestamp": "2020-02-01T17:15:08.9284049Z",
"ackVersion": 3
},
"fwVersion": { "ackVersion": 3 },
"status": {
"desiredValue": {
"first": "first",
"second": "second"
},
"desiredVersion": 2,
"desiredTimestamp": "2020-02-01T17:15:08.9284049Z",
"ackVersion": 2
}
},
}
},
"installDate": { "installDate": "2020-02-01" }
}
Device templates
Each message or snapshot record represents one or more changes to a published device template since the last exported message. Information sent in each message or record includes:
idof the device template that matches theinstanceOfof the devices stream abovedisplayNameof the device template- The device
capabilityModelincluding itsinterfaces, and the telemetry, properties, and commands definitions cloudPropertiesdefinitions- Overrides and initial values, inline with the
capabilityModel
Deleted device templates aren't exported. Currently, there are no indicators in exported messages for deleted device templates.
For Event Hubs and Service Bus, IoT Central sends messages containing device template data to your event hub or Service Bus queue or topic in near real time.
For Blob storage, a new snapshot containing all the changes since the last one written is exported once per minute.
This example shows a message about device templates data in event hub or Service Bus queue or topic:
{
"body":{
"id": "<device template id>",
"etag": "<etag>",
"types": ["DeviceModel"],
"displayName": "Sensor template",
"capabilityModel": {
"@id": "<capability model id>",
"@type": ["CapabilityModel"],
"contents": [],
"implements": [
{
"@id": "<component Id>",
"@type": ["InterfaceInstance"],
"name": "sensorComponent",
"schema": {
"@id": "<interface Id>",
"@type": ["Interface"],
"displayName": "Sensor interface",
"contents": [
{
"@id": "<id>",
"@type": ["Telemetry"],
"displayName": "Humidity",
"name": "humidity",
"schema": "double"
},
{
"@id": "<id>",
"@type": ["Telemetry", "SemanticType/Event"],
"displayName": "Error event",
"name": "error",
"schema": "integer"
},
{
"@id": "<id>",
"@type": ["Property"],
"displayName": "Set temperature",
"name": "setTemp",
"writable": true,
"schema": "integer",
"unit": "Units/Temperature/fahrenheit",
"initialValue": "30"
},
{
"@id": "<id>",
"@type": ["Property"],
"displayName": "Firmware version read only",
"name": "fwversion",
"schema": "string"
},
{
"@id": "<id>",
"@type": ["Property"],
"displayName": "Display status",
"name": "status",
"writable": true,
"schema": {
"@id": "urn:testInterface:status:obj:ka8iw8wka:1",
"@type": ["Object"]
}
},
{
"@id": "<id>",
"@type": ["Command"],
"request": {
"@id": "<id>",
"@type": ["SchemaField"],
"displayName": "Configuration",
"name": "config",
"schema": "string"
},
"response": {
"@id": "<id>",
"@type": ["SchemaField"],
"displayName": "Response",
"name": "response",
"schema": "string"
},
"displayName": "Configure sensor",
"name": "sensorConfig"
}
]
}
}
],
"displayName": "Sensor capability model"
},
"solutionModel": {
"@id": "<id>",
"@type": ["SolutionModel"],
"cloudProperties": [
{
"@id": "<id>",
"@type": ["CloudProperty"],
"displayName": "Install date",
"name": "installDate",
"schema": "dateTime",
"valueDetail": {
"@id": "<id>",
"@type": ["ValueDetail/DateTimeValueDetail"]
}
}
]
}
},
"annotations":{
"iotcentral-message-source":"deviceTemplates",
"x-opt-partition-key":"<partitionKey>",
"x-opt-sequence-number":25315,
"x-opt-offset":"<offset>",
"x-opt-enqueued-time":1539274985085
},
"partitionKey":"<partitionKey>",
"sequenceNumber":25315,
"enqueuedTimeUtc":"2019-10-02T16:23:05.085Z",
"offset":"<offset>"
}
}
This example snapshot shows a message that contains device and properties data in Blob storage. Exported files contain a single line per record.
{
"id": "<device template id>",
"etag": "<etag>",
"types": ["DeviceModel"],
"displayName": "Sensor template",
"capabilityModel": {
"@id": "<capability model id>",
"@type": ["CapabilityModel"],
"contents": [],
"implements": [
{
"@id": "<component Id>",
"@type": ["InterfaceInstance"],
"name": "Sensor component",
"schema": {
"@id": "<interface Id>",
"@type": ["Interface"],
"displayName": "Sensor interface",
"contents": [
{
"@id": "<id>",
"@type": ["Telemetry"],
"displayName": "Humidity",
"name": "humidity",
"schema": "double"
},
{
"@id": "<id>",
"@type": ["Telemetry", "SemanticType/Event"],
"displayName": "Error event",
"name": "error",
"schema": "integer"
},
{
"@id": "<id>",
"@type": ["Property"],
"displayName": "Set temperature",
"name": "setTemp",
"writable": true,
"schema": "integer",
"unit": "Units/Temperature/fahrenheit",
"initialValue": "30"
},
{
"@id": "<id>",
"@type": ["Property"],
"displayName": "Firmware version read only",
"name": "fwversion",
"schema": "string"
},
{
"@id": "<id>",
"@type": ["Property"],
"displayName": "Display status",
"name": "status",
"writable": true,
"schema": {
"@id": "urn:testInterface:status:obj:ka8iw8wka:1",
"@type": ["Object"]
}
},
{
"@id": "<id>",
"@type": ["Command"],
"request": {
"@id": "<id>",
"@type": ["SchemaField"],
"displayName": "Configuration",
"name": "config",
"schema": "string"
},
"response": {
"@id": "<id>",
"@type": ["SchemaField"],
"displayName": "Response",
"name": "response",
"schema": "string"
},
"displayName": "Configure sensor",
"name": "sensorconfig"
}
]
}
}
],
"displayName": "Sensor capability model"
},
"solutionModel": {
"@id": "<id>",
"@type": ["SolutionModel"],
"cloudProperties": [
{
"@id": "<id>",
"@type": ["CloudProperty"],
"displayName": "Install date",
"name": "installDate",
"schema": "dateTime",
"valueDetail": {
"@id": "<id>",
"@type": ["ValueDetail/DateTimeValueDetail"]
}
}
]
}
}
Data format change notice
Note
The telemetry stream data format is unaffected by this change. Only the devices and device templates streams of data are affected.
If you have an existing data export in your preview application with the Devices and Device templates streams turned on, update your export by 30 June 2020. This requirement applies to exports to Azure Blob storage, Azure Event Hubs, and Azure Service Bus.
Starting 3 February 2020, all new exports in applications with Devices and Device templates enabled will have the data format described above. All exports created before this date remain on the old data format until 30 June 2020, at which time these exports will automatically be migrated to the new data format. The new data format matches the device, device property, and device template objects in the IoT Central public API.
For Devices, notable differences between the old data format and the new data format include:
@idfor device is removed,deviceIdis renamed toidprovisionedflag is added to describe the provisioning status of the deviceapprovedflag is added to describe the approval state of the devicepropertiesincluding device and cloud properties, matches entities in the public API
For Device templates, notable differences between the old data format and the new data format include:
@idfor device template is renamed toid@typefor the device template is renamed totypes, and is now an array
Devices (format deprecated as of 3 February 2020)
{
"@id":"<id-value>",
"@type":"Device",
"displayName":"Airbox",
"data":{
"$cloudProperties":{
"Color":"blue"
},
"EnvironmentalSensor":{
"thsensormodel":{
"reported":{
"value":"Neque quia et voluptatem veritatis assumenda consequuntur quod.",
"$lastUpdatedTimestamp":"2019-09-30T20:35:43.8478978Z"
}
},
"pm25sensormodel":{
"reported":{
"value":"Aut alias odio.",
"$lastUpdatedTimestamp":"2019-09-30T20:35:43.8478978Z"
}
}
},
"urn_azureiot_DeviceManagement_DeviceInformation":{
"totalStorage":{
"reported":{
"value":27900.9730905171,
"$lastUpdatedTimestamp":"2019-09-30T20:35:43.8478978Z"
}
},
"totalMemory":{
"reported":{
"value":4667.82916715811,
"$lastUpdatedTimestamp":"2019-09-30T20:35:43.8478978Z"
}
}
}
},
"instanceOf":"<template-id>",
"deviceId":"<device-id>",
"simulated":true
}
Device templates (format deprecated as of 3 February 2020)
{
"@id":"<template-id>",
"@type":"DeviceModelDefinition",
"displayName":"Airbox",
"capabilityModel":{
"@id":"<id>",
"@type":"CapabilityModel",
"implements":[
{
"@id":"<id>",
"@type":"InterfaceInstance",
"name":"EnvironmentalSensor",
"schema":{
"@id":"<id>",
"@type":"Interface",
"comment":"Requires temperature and humidity sensors.",
"description":"Provides functionality to report temperature, humidity. Provides telemetry, commands and read-write properties",
"displayName":"Environmental Sensor",
"contents":[
{
"@id":"<id>",
"@type":"Telemetry",
"description":"Current temperature on the device",
"displayName":"Temperature",
"name":"temp",
"schema":"double",
"unit":"Units/Temperature/celsius",
"valueDetail":{
"@id":"<id>",
"@type":"ValueDetail/NumberValueDetail",
"minValue":{
"@value":"50"
}
},
"visualizationDetail":{
"@id":"<id>",
"@type":"VisualizationDetail"
}
},
{
"@id":"<id>",
"@type":"Telemetry",
"description":"Current humidity on the device",
"displayName":"Humidity",
"name":"humid",
"schema":"integer"
},
{
"@id":"<id>",
"@type":"Telemetry",
"description":"Current PM2.5 on the device",
"displayName":"PM2.5",
"name":"pm25",
"schema":"integer"
},
{
"@id":"<id>",
"@type":"Property",
"description":"T&H Sensor Model Name",
"displayName":"T&H Sensor Model",
"name":"thsensormodel",
"schema":"string"
},
{
"@id":"<id>",
"@type":"Property",
"description":"PM2.5 Sensor Model Name",
"displayName":"PM2.5 Sensor Model",
"name":"pm25sensormodel",
"schema":"string"
}
]
}
},
{
"@id":"<id>",
"@type":"InterfaceInstance",
"name":"urn_azureiot_DeviceManagement_DeviceInformation",
"schema":{
"@id":"<id>",
"@type":"Interface",
"displayName":"Device information",
"contents":[
{
"@id":"<id>",
"@type":"Property",
"comment":"Total available storage on the device in kilobytes. Ex. 20480000 kilobytes.",
"displayName":"Total storage",
"name":"totalStorage",
"displayUnit":"kilobytes",
"schema":"long"
},
{
"@id":"<id>",
"@type":"Property",
"comment":"Total available memory on the device in kilobytes. Ex. 256000 kilobytes.",
"displayName":"Total memory",
"name":"totalMemory",
"displayUnit":"kilobytes",
"schema":"long"
}
]
}
}
],
"displayName":"AAEONAirbox52"
},
"solutionModel":{
"@id":"<id>",
"@type":"SolutionModel",
"cloudProperties":[
{
"@id":"<id>",
"@type":"CloudProperty",
"displayName":"Color",
"name":"Color",
"schema":"string",
"valueDetail":{
"@id":"<id>",
"@type":"ValueDetail/StringValueDetail"
},
"visualizationDetail":{
"@id":"<id>",
"@type":"VisualizationDetail"
}
}
]
}
}
Next steps
Now that you know how to export your data to Azure Event Hubs, Azure Service Bus, and Azure Blob storage, continue to the next step:
Feedback
Submit and view feedback for