Tutorial: React to Blob Storage events on IoT Edge (Preview)
This article shows you how to deploy the Azure Blob Storage on IoT module, which would act as an Event Grid publisher to send events on Blob creation and Blob deletion to Event Grid.
For an overview of the Azure Blob Storage on IoT Edge, see Azure Blob Storage on IoT Edge and its features.
Important
On March 31, 2023, Event Grid on Azure IoT Edge support will be retired, so make sure to transition to IoT Edge native capabilities prior to that date. For more information, see Transition from Event Grid on Azure IoT Edge to Azure IoT Edge.
Warning
Azure Blob Storage on IoT Edge integration with Event Grid is in Preview
In order to complete this tutorial, you will need:
- Azure subscription - Create a free account if you don't already have one.
- Azure IoT Hub and IoT Edge device - Follow the steps in the quickstart for Linux or Windows devices if you don't already have one.
Deploy Event Grid IoT Edge module
There are several ways to deploy modules to an IoT Edge device and all of them work for Azure Event Grid on IoT Edge. This article describes the steps to deploy Event Grid on IoT Edge from the Azure portal.
Note
In this tutorial, you will deploy the Event Grid module without persistence. It means that any topics and subscriptions you create in this tutorial will be deleted when you redeploy the module. For more information on how to setup persistence, see the following articles: Persist state in Linux or Persist state in Windows. For production workloads, we recommend that you install the Event Grid module with persistence.
Select your IoT Edge device
- Sign in to the Azure portal
- Navigate to your IoT Hub.
- Select IoT Edge from the menu in the Automatic Device Management section.
- Click on the ID of the target device from the list of devices
- Select Set Modules. Keep the page open. You will continue with the steps in the next section.
Configure a deployment manifest
A deployment manifest is a JSON document that describes which modules to deploy, how data flows between the modules, and desired properties of the module twins. The Azure portal has a wizard that walks you through creating a deployment manifest, instead of building the JSON document manually. It has three steps: Add modules, Specify routes, and Review deployment.
Add modules
In the Deployment Modules section, select Add
From the types of modules in the drop-down list, select IoT Edge Module
Provide the name, image, container create options of the container:
- Name: eventgridmodule
- Image URI:
mcr.microsoft.com/azure-event-grid/iotedge:latest
- Container Create Options:
{ "Env": [ "inbound__serverAuth__tlsPolicy=enabled", "inbound__clientAuth__clientCert__enabled=false" ], "HostConfig": { "PortBindings": { "4438/tcp": [ { "HostPort": "4438" } ] } } }
Click Save
Continue to the next section to add the Azure Event Grid Subscriber module before deploying them together.
Important
In this tutorial, you will learn to deploy the Event Grid module to allow both HTTP/HTTPs requests, client authentication disabled. For production workloads, we recommend that you enable only HTTPs requests and subscribers with client authentication enabled. For more information on how to configure Event Grid module securely, see Security and authentication.
Deploy Event Grid Subscriber IoT Edge module
This section shows you how to deploy another IoT module which would act as an event handler to which events can be delivered.
Add modules
In the Deployment Modules section, select Add again.
From the types of modules in the drop-down list, select IoT Edge Module
Provide the name, image, and container create options of the container:
- Name: subscriber
- Image URI:
mcr.microsoft.com/azure-event-grid/iotedge-samplesubscriber:latest
- Container Create Options: None
Click Save
Continue to the next section to add the Azure Blob Storage module
Deploy Azure Blob Storage module
This section shows you how to deploy the Azure Blob Storage module, which would act as an Event Grid publisher publishing Blob created and deleted events.
Add modules
In the Deployment Modules section, select Add
From the types of modules in the drop-down list, select IoT Edge Module
Provide the name, image, and container create options of the container:
- Name: azureblobstorageoniotedge
- Image URI: mcr.microsoft.com/azure-blob-storage:latest
- Container Create Options:
{ "Env":[ "LOCAL_STORAGE_ACCOUNT_NAME=<your storage account name>", "LOCAL_STORAGE_ACCOUNT_KEY=<your storage account key>", "EVENTGRID_ENDPOINT=http://<event grid module name>:5888" ], "HostConfig":{ "Binds":[ "<storage mount>" ], "PortBindings":{ "11002/tcp":[{"HostPort":"11002"}] } } }
Important
- Blob Storage module can publish events using both HTTPS and HTTP.
- If you have enabled the client-based authentication for EventGrid, make sure you update the value of EVENTGRID_ENDPOINT to allow https, like this:
EVENTGRID_ENDPOINT=https://<event grid module name>:4438
. - Also add another environment variable
AllowUnknownCertificateAuthority=true
to the above Json. When talking to EventGrid over HTTPS, AllowUnknownCertificateAuthority allows the storage module to trust self-signed EventGrid server certificates.
Update the JSON that you copied with the following information:
Replace
<your storage account name>
with a name that you can remember. Account names should be 3 to 24 characters long, with lowercase letters and numbers. No spaces.Replace
<your storage account key>
with a 64-byte base64 key. You can generate a key with tools like GeneratePlus. You'll use these credentials to access the blob storage from other modules.Replace
<event grid module name>
with the name of your Event Grid module.Replace
<storage mount>
according to your container operating system.- For Linux containers, my-volume:/blobroot
- For Windows containers,my-volume:C:/BlobRoot
Click Save
Click Next to continue to the routes section
Note
If you are using an Azure VM as the edge device, add an inbound port rule to allow inbound traffic on the host ports used in this tutorial: 4438, 5888, 8080, and 11002. For instructions on adding the rule, see How to open ports to a VM.
Setup routes
Keep the default routes, and select Next to continue to the review section
Review deployment
- The review section shows you the JSON deployment manifest that was created based on your selections in the previous section. Confirm that you see the following four modules: $edgeAgent, $edgeHub, eventgridmodule, subscriber and azureblobstorageoniotedge that all being deployed.
- Review your deployment information, then select Submit.
Verify your deployment
After you submit the deployment, you return to the IoT Edge page of your IoT hub.
Select the IoT Edge device that you targeted with the deployment to open its details.
In the device details, verify that the eventgridmodule, subscriber and azureblobstorageoniotedge modules are listed as both Specified in deployment and Reported by device.
It may take a few moments for the module to be started on the device and then reported back to IoT Hub. Refresh the page to see an updated status.
Publish BlobCreated and BlobDeleted events
This module automatically creates topic MicrosoftStorage. Verify that it exists
curl -k -H "Content-Type: application/json" -X GET -g https://<your-edge-device-public-ip-here>:4438/topics/MicrosoftStorage?api-version=2019-01-01-preview
Sample output:
[ { "id": "/iotHubs/eg-iot-edge-hub/devices/eg-edge-device/modules/eventgridmodule/topics/MicrosoftStorage", "name": "MicrosoftStorage", "type": "Microsoft.EventGrid/topics", "properties": { "endpoint": "https://<edge-vm-ip>:4438/topics/MicrosoftStorage/events?api-version=2019-01-01-preview", "inputSchema": "EventGridSchema" } } ]
Important
- For the HTTPS flow, if the client authentication is enabled via SAS key, then the SAS key specified earlier should be added as a header. Hence the curl request will be:
curl -k -H "Content-Type: application/json" -H "aeg-sas-key: <your SAS key>" -X GET -g https://<your-edge-device-public-ip-here>:4438/topics/MicrosoftStorage?api-version=2019-01-01-preview
- For the HTTPS flow, if the client authentication is enabled via certificate, the curl request will be:
curl -k -H "Content-Type: application/json" --cert <certificate file> --key <certificate private key file> -X GET -g https://<your-edge-device-public-ip-here>:4438/topics/MicrosoftStorage?api-version=2019-01-01-preview
- For the HTTPS flow, if the client authentication is enabled via SAS key, then the SAS key specified earlier should be added as a header. Hence the curl request will be:
Subscribers can register for events published to a topic. To receive any event, you'll need to create an Event Grid subscription for MicrosoftStorage topic.
Create blobsubscription.json with the following content. For details about the payload, see our API documentation
{ "properties": { "destination": { "endpointType": "WebHook", "properties": { "endpointUrl": "https://subscriber:4430" } } } }
Note
The endpointType property specifies that the subscriber is a Webhook. The endpointUrl specifies the URL at which the subscriber is listening for events. This URL corresponds to the Azure Function sample you deployed earlier.
Run the following command to create a subscription for the topic. Confirm that you see the HTTP status code is
200 OK
.curl -k -H "Content-Type: application/json" -X PUT -g -d @blobsubscription.json https://<your-edge-device-public-ip-here>:4438/topics/MicrosoftStorage/eventSubscriptions/sampleSubscription5?api-version=2019-01-01-preview
Important
- For the HTTPS flow, if the client authentication is enabled via SAS key, then the SAS key specified earlier should be added as a header. Hence the curl request will be:
curl -k -H "Content-Type: application/json" -H "aeg-sas-key: <your SAS key>" -X PUT -g -d @blobsubscription.json https://<your-edge-device-public-ip-here>:4438/topics/MicrosoftStorage/eventSubscriptions/sampleSubscription5?api-version=2019-01-01-preview
- For the HTTPS flow, if the client authentication is enabled via certificate, the curl request will be:
curl -k -H "Content-Type: application/json" --cert <certificate file> --key <certificate private key file> -X PUT -g -d @blobsubscription.json https://<your-edge-device-public-ip-here>:4438/topics/MicrosoftStorage/eventSubscriptions/sampleSubscription5?api-version=2019-01-01-preview
- For the HTTPS flow, if the client authentication is enabled via SAS key, then the SAS key specified earlier should be added as a header. Hence the curl request will be:
Run the following command to verify subscription was created successfully. HTTP Status Code of 200 OK should be returned.
curl -k -H "Content-Type: application/json" -X GET -g https://<your-edge-device-public-ip-here>:4438/topics/MicrosoftStorage/eventSubscriptions/sampleSubscription5?api-version=2019-01-01-preview
Sample output:
{ "id": "/iotHubs/eg-iot-edge-hub/devices/eg-edge-device/modules/eventgridmodule/topics/MicrosoftStorage/eventSubscriptions/sampleSubscription5", "type": "Microsoft.EventGrid/eventSubscriptions", "name": "sampleSubscription5", "properties": { "Topic": "MicrosoftStorage", "destination": { "endpointType": "WebHook", "properties": { "endpointUrl": "https://subscriber:4430" } } } }
Important
- For the HTTPS flow, if the client authentication is enabled via SAS key, then the SAS key specified earlier should be added as a header. Hence the curl request will be:
curl -k -H "Content-Type: application/json" -H "aeg-sas-key: <your SAS key>" -X GET -g https://<your-edge-device-public-ip-here>:4438/topics/MicrosoftStorage/eventSubscriptions/sampleSubscription5?api-version=2019-01-01-preview
- For the HTTPS flow, if the client authentication is enabled via certificate, the curl request will be:
curl -k -H "Content-Type: application/json" --cert <certificate file> --key <certificate private key file> -X GET -g https://<your-edge-device-public-ip-here>:4438/topics/MicrosoftStorage/eventSubscriptions/sampleSubscription5?api-version=2019-01-01-preview
- For the HTTPS flow, if the client authentication is enabled via SAS key, then the SAS key specified earlier should be added as a header. Hence the curl request will be:
Download Azure Storage Explorer and connect it to your local storage
Verify event delivery
Verify BlobCreated event delivery
Upload files as block blobs to the local storage from Azure Storage Explorer, and the module will automatically publish create events.
Check out the subscriber logs for create event. Follow the steps to verify the event delivery
Sample Output:
Received Event: { "id": "d278f2aa-2558-41aa-816b-e6d8cc8fa140", "topic": "MicrosoftStorage", "subject": "/blobServices/default/containers/cont1/blobs/Team.jpg", "eventType": "Microsoft.Storage.BlobCreated", "eventTime": "2019-10-01T21:35:17.7219554Z", "dataVersion": "1.0", "metadataVersion": "1", "data": { "api": "PutBlob", "clientRequestId": "00000000-0000-0000-0000-000000000000", "requestId": "ef1c387b-4c3c-4ac0-8e04-ff73c859bfdc", "eTag": "0x8D746B740DA21FB", "url": "http://azureblobstorageoniotedge:11002/myaccount/cont1/Team.jpg", "contentType": "image/jpeg", "contentLength": 858129, "blobType": "BlockBlob" } }
Verify BlobDeleted event delivery
Delete blobs from the local storage using Azure Storage Explorer, and the module will automatically publish delete events.
Check out the subscriber logs for delete event. Follow the steps to verify the event delivery
Sample Output:
Received Event: { "id": "ac669b6f-8b0a-41f3-a6be-812a3ce6ac6d", "topic": "MicrosoftStorage", "subject": "/blobServices/default/containers/cont1/blobs/Team.jpg", "eventType": "Microsoft.Storage.BlobDeleted", "eventTime": "2019-10-01T21:36:09.2562941Z", "dataVersion": "1.0", "metadataVersion": "1", "data": { "api": "DeleteBlob", "clientRequestId": "00000000-0000-0000-0000-000000000000", "requestId": "2996bbfb-c819-4d02-92b1-c468cc67d8c6", "eTag": "0x8D746B740DA21FB", "url": "http://azureblobstorageoniotedge:11002/myaccount/cont1/Team.jpg", "contentType": "image/jpeg", "contentLength": 858129, "blobType": "BlockBlob" } }
Congratulations! You have completed the tutorial. The following sections provide details on the event properties.
Event properties
Here's the list of supported event properties and their types and descriptions.
Property | Type | Description |
---|---|---|
topic | string | Full resource path to the event source. This field is not writeable. Event Grid provides this value. |
subject | string | Publisher-defined path to the event subject. |
eventType | string | One of the registered event types for this event source. |
eventTime | string | The time the event is generated based on the provider's UTC time. |
id | string | Unique identifier for the event. |
data | object | Blob storage event data. |
dataVersion | string | The schema version of the data object. The publisher defines the schema version. |
metadataVersion | string | The schema version of the event metadata. Event Grid defines the schema of the top-level properties. Event Grid provides this value. |
The data object has the following properties:
Property | Type | Description |
---|---|---|
api | string | The operation that triggered the event. It can be one of the following values:
|
clientRequestId | string | a client-provided request ID for the storage API operation. This ID can be used to correlate to Azure Storage diagnostic logs using the "client-request-id" field in the logs, and can be provided in client requests using the "x-ms-client-request-id" header. For details, see Log Format. |
requestId | string | Service-generated request ID for the storage API operation. Can be used to correlate to Azure Storage diagnostic logs using the "request-id-header" field in the logs and is returned from initiating API call in the 'x-ms-request-id' header. See Log Format. |
eTag | string | The value that you can use to perform operations conditionally. |
contentType | string | The content type specified for the blob. |
contentLength | integer | The size of the blob in bytes. |
blobType | string | The type of blob. Valid values are either "BlockBlob" or "PageBlob". |
url | string | The path to the blob. If the client uses a Blob REST API, then the url has this structure: <storage-account-name>.blob.core.windows.net/<container-name>/<file-name>. If the client uses a Data Lake Storage REST API, then the url has this structure: <storage-account-name>.dfs.core.windows.net/<file-system-name>/<file-name>. |
Next steps
See the following articles from the Blob Storage documentation:
In this tutorial, you published events by creating or deleting blobs in an Azure Blob Storage. See the other tutorials to learn how to forward events to cloud (Azure Event Hub or Azure IoT Hub):
Feedback
Submit and view feedback for