question

ChrisU-9274 avatar image
0 Votes"
ChrisU-9274 asked BrunoLucas-9843 edited

IoT Hub and Azure Functions using Python and Visual Studio Code

To set the scene: I am new to coding and in my first coding/software support role having completed a diploma last year to bring about a career change. I am also new to IoT but have been tasked with delivering a IoT proof of concept to my employer. So far, I have managed to setup a couple of gateways and devices on a LoRaWAN network server, output to an Azure IoT Hub and store the data in an Azure SQL DB using a Stream analytics job.

However, the sensor environment data arrives encoded in hexadecimal format and so needs decoding. I have already set up a WebSocket output and a simple app that consumes and decodes the data to variables that you can then use/store. This is within Visual Studio Code and run from the console/terminal; I have not yet been able to create an Azure Function for this. I want to now use this decoding script I have written within an Azure function which will take the uplink data, decode the environment data, and then output these along with other data to an Azure database. The trigger being the receiving of uplink data within the IoT-Hub:

198546-image.png


In this, the decode function would replace the Stream analytics job. Would this work or would I require additional functionality?

From reading the documentation here, I think I am right in that an IoT-Hub is an Event hub? I am using Visual Studio Code and Python and I have found very little documentation on setting up Azure Functions for IoT Hub/Event Hub and those that I have found focus on C# and Visual studio. Is there any other resource that I can be pointed to that may help?

Thanks in advance


azure-functionsazure-iot-hubazure-event-hubs
image.png (91.8 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I guess some of the confusion is, IOT hub is an Event Hub structured for IOT, but nevertheless different from one each other in terms of features.

What works for both is the same trigger:

198811-image.png

There is a trigger for IoT for older .net versions. I don't see it anymore if you select the most recent .net version. Even the documentation refers to the EventHubTrigger under the iothub doco: https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-iot-trigger?tabs=in-process%2Cfunctionsv2%2Cextensionv5&pivots=programming-language-csharp

198719-image.png


1 Vote 1 ·
image.png (7.5 KiB)
BrunoLucas-9843 avatar image
0 Votes"
BrunoLucas-9843 answered BrunoLucas-9843 edited

Hi @ChrisU-9274 ,

I cloned your code and it seems correct. I managed to debug it and deploy it.

First. the debugging locally problem could be 2 things:

1 - Firewall. the trigger function defaults to AMQP, so, it is trying to reach tpc ports 5671 and 5672. usually I fix this by creating a firewall rule but I see couple months ago Microsoft has finally provided the option to change it to amqpwebsocket. First try adjusting the firewall. if doesn't work, try to change your host.json to this:

 {
   "version": "2.0",
   "extensions": {
     "eventHubs": {
       "transportType" : "amqpWebSockets"
     }
   },
   "logging": {
     "applicationInsights": {
       "samplingSettings": {
         "isEnabled": true,
         "excludedTypes": "Request"
       }
     }
   },
   "extensionBundle": {
     "id": "Microsoft.Azure.Functions.ExtensionBundle",
     "version": "[2.*, 3.0.0)"
   }
 }

2 - Did your VS Code generated the python environment? if so, you should have a folder called ".venv"
200961-image.png

Regarding deployment, did you use the vs code deploy?
200972-image.png

I think part of it is an add-on bug. you can try to deploy without an ide: https://docs.microsoft.com/en-us/azure/azure-functions/deployment-zip-push if you prefer, but back to vs code, for now, after deployment, just make sure the azure function settings (in the cloud) match the ones on the app.seetings.json. the hub connection string must be the same name in Azure. did you grab that here?:

201023-image.png

The storage is not mapped in the bindings but seems to be picked as long the name is "AzureWebJobsStorage". just need to make sure the value is correct.

200914-image.png

Before moving into testing (and after deploying) remember you need to stop the azure function in azure when testing locally, otherwise the cloud will always grab the messages.

200992-image.png

To send messages for testing, I use the iot hub add-on for vs code:

https://marketplace.visualstudio.com/items?itemName=vsciot-vscode.azure-iot-tools
https://marketplace.visualstudio.com/items?itemName=vsciot-vscode.azure-iot-toolkit

200910-image.png
200936-image.png

Did you add any devices? the vs code message sending add-on will expect at least one device. If you don't, just add on under the iot hub in azure before using the add-on

The right place to check if that is running in azure is here:


Functions > [the function you want to check] > Monitor:


200968-image.png



200987-image.png



the logs are good for initial troubleshoot. this line writes to that log:
logging.info('Python EventHub trigger processed an event: %s',
event.get_body().decode('utf-8'))

200963-image.png

to debug locally, make sure the azure function is stopped. hit f5 or click here :

200930-image.png

Remember to install this if you want to use the iot hub add-on: https://marketplace.visualstudio.com/items?itemName=vsciot-vscode.azure-iot-tools . It helps testing/debugging

The confusion in VS Code is where to go to use add-ons. if you were in the devug section, to find the "send message" you need to click in the explorer:

200944-image.png

you should see this:

200966-image.png
200985-image.png
201002-image.png

If everything is good, you should see this:
200918-image.png


Would be good to first debug locally to confirm you have the correct connection strings.

Let me know if that helps.



image.png (5.5 KiB)
image.png (23.4 KiB)
image.png (142.9 KiB)
image.png (14.0 KiB)
image.png (30.4 KiB)
image.png (45.6 KiB)
image.png (28.6 KiB)
image.png (22.9 KiB)
image.png (36.3 KiB)
image.png (21.9 KiB)
image.png (29.9 KiB)
image.png (27.1 KiB)
image.png (8.9 KiB)
image.png (8.2 KiB)
image.png (40.4 KiB)
image.png (14.0 KiB)
image.png (34.5 KiB)
image.png (90.1 KiB)
image.png (46.4 KiB)
image.png (48.6 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Bruno, thanks for your comprehensive response! I have been working on other issues today and only just reading this now. I will work through this tomorrow more thoroughly. However just quickly, in regards to the functions monitoring, if it is stopped if get the below:
201151-image.png

if it is running I get this, so my function is not listed:
201100-image.png


0 Votes 0 ·
image.png (31.6 KiB)
image.png (30.3 KiB)

Hi @ChrisU-9274 ,

This could be the connection string to the storage is wrong (maybe blocked) or something didn't deploy.
Check to see if you have the correct storage connection string first. I covered that on my last answer. If that is correct, check to see if the code is deployed. you can check that here:

201291-image.png

switch to function,json to see if the binding is also correct

201227-image.png

Did you create the iot hub and the storage? Did you add any vpn or white list filter to it after creating?




Could you vote on the previous answers if the info have helped? That also help other forum members to find answers.
Cheers!


0 Votes 0 ·
image.png (46.6 KiB)
MatthijsvdVeer avatar image
1 Vote"
MatthijsvdVeer answered ChrisU-9274 commented

You're on the right track! You could definitely replace Stream Analytics with an Azure Function, you could even use both in series if you wanted, but it seems like using the Function might be enough for your use case. Here's a link to an Event-Hub triggered Azure Function for Python. If you're not sure how to set up an Azure Function is VS Code, you can download this extension. The extension has a command to start a new Azure Functions project, which will ask you what binding you need:

198724-image.png

You will need to grab the Event Hub endpoint connection string from the portal to connect it up.
198725-image.png

You mentioned

an IoT-Hub is an Event hub?

That's not exactly right, an IoT Hub does expose a default Event Hub endpoint that you can grab messages off. But that Event Hub is only a small part of IoT Hub. In any case you can definitely use an Event Hub trigger in an Azure Function to retrieve the messages!


image.png (73.6 KiB)
image.png (174.1 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you for your response, I have continued my work on this to try and progress my understanding. In regards to your answer, where you state 'You will need to grab the Event Hub endpoint connection string from the portal to connect it up.', do you mean that it is entered as below into the function.json file:

198880-image.png

Also, what I think is not clear, is where I define the trigger condition. I want my trigger to be each time an uplink is received. Where is this defined for instance in the code examples in the link in your response (here)?


0 Votes 0 ·
image.png (27.1 KiB)

I also want to check my understanding of the 'trigger' part of an event hub trigger, as thinking about it now, I suspect that my understanding is incorrect. If we assume that the IoT-Hub is the event hub then am I right in that the Azure Event Hub trigger function that is created will consume the data every time and depending on conditional statements within the trigger, determine if the data is passed to other resources? So for instance, a conditional statement that will only store temperature if it is over 25 degrees, otherwise it is ignored.

So where I have stated I want the uplink message to be the trigger; is this incorrect and the uplink data be consumed each time? And the trigger is the other resources that are engaged depending on the data received?

If so, the first thing I would want to do is output the data received. So using python, grab this data and log it to the console to confirm it is received and in the format I expect.

0 Votes 0 ·
ChrisU-9274 avatar image
0 Votes"
ChrisU-9274 answered

So I am still struggling on with this. I have watched quite a few tutorials and read some more of the documentation and therefore understand a bit more. I decided to try and follow the documentation for the Azure IoT bindings and triggers and run it to see what will happen and what gets logged. However, I have not got even that far and wondering whether anyone can take a look and point out where I am going wrong.

The issue is below, I set this within function.json and took the value from that displayed within the built in endpoints:
199669-image.png

My code:
inti.py:
199680-image.png

function.json:
199568-image.png

host.json:
199723-image.png

The connection binding was taken from here:
199731-image.png



image.png (107.4 KiB)
image.png (36.5 KiB)
image.png (25.4 KiB)
image.png (58.3 KiB)
image.png (83.2 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

BrunoLucas-9843 avatar image
0 Votes"
BrunoLucas-9843 answered BrunoLucas-9843 commented

Hi @ChrisU-9274 ,

You need to add a azure storage to it. that error appear to be you didn't add the conn strings to the az function settings. It won't pick up from the code file.

Did you manage to run that using the visual studio debugger?

I managed to debug and deploy. I committed my code here:

https://github.com/blucas2016/PythonIoTTriggerSampleCode

clone that and give it a try.

I generated the code above using the vs code az function add-on. (you may want to close and restart vs code after installing those add-ons)
199846-image.png

If you run that it will guide you through setting up the app setting file (I recommend you create your azure storage before so you can choose when it prompt you to select storage ):
199853-image.png

That will generate something like this:

199876-image.png

You can see it added a local.settings.json with (#1). The azure storage conn string - (#2). the event hub connection string. When I run the debugger (after sending events to the hub) I do see it coming though (#3)

Git won't commit "local.settings.json", add this to your solution if you need (add your own conn strings)

 {
   "IsEncrypted": false,
   "Values": {
     "AzureWebJobsStorage": "DefaultEnxxointsProtoxxxxxxxxxxxxxxxxxx=f5KL1VKGwPL+CJQwsIsIASx57xxxxxxxxxxmHhxxxxxxxxdows.net",
     "FUNCTIONS_WORKER_RUNTIME": "python",
     "brunoqnadevhub_RootManageSharedAccessKey_EVENTHUB": "Enxpxxoint=sb:/xxxxxxxx20-18914xb4b61d8xxxxxxxxx;SharedAccessKey=gMZSeO70EELLzbAv31kRexxxx022"
   }
      
 }

The hub conn string is the one from "built-in endpoints":

199883-image.png

And the storage is the connection string:

199787-image.png

Now that i now the code works, it would make easier to deploy and narrow down possible errors. I used the az function deployment button from the vs code add on:

199737-image.png


199845-image.png

For some reason the vs code deployment added (#1) but skipped the hub conn (#2) I manually added direct in azure

after sending some messages I can now confirm deployment is successful

199881-image.png

Let me know if that helps. I can try to add more details . If that is enough, please Accept the answer. Cheers!



image.png (16.0 KiB)
image.png (67.2 KiB)
image.png (77.9 KiB)
image.png (13.8 KiB)
image.png (46.9 KiB)
image.png (52.8 KiB)
image.png (30.3 KiB)
image.png (9.8 KiB)
image.png (12.3 KiB)
image.png (7.7 KiB)
image.png (13.2 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks for the response Bruno. Will work through this today and tomorrow and post back anything.

0 Votes 0 ·

Forgot to add that azure hub trigger functions use the AMQP protocol by default and many devs may get this error or not depending on their machine security: https://brunolucas.blog/2022/02/27/code-and-test-azure-service-bus-and-azure-event-hub-triggers-locally/
That may affect debugging locally but should not be a problem when you deploy to azure

1 Vote 1 ·
ChrisU-9274 avatar image
0 Votes"
ChrisU-9274 answered

Bruno, so I have sought to replicate what you did above. Thank you for pointing out the need to define the storage and showing me how. Anyway, as you suggested I set up an eventhubtrigger connecting to my IoT Hub and replicated what you did. I did not change the init.py or the json files as suggested here and here.

My repo is here.

The local.settings.json:
200597-image.png

When tapping F5 to run locally I got the following in the terminal:
200548-image.png


Can you explain this?

I did however upload to Azure where, like you I had to add the event hub connection key and value. What was weird was that the AzureWebJobsStorage Value was different to that I had set within the local.settings.json. I decide to run it with this value and then with the value within the local.settings.json.

Run with initial AzureWebJobsStorage value:
Started the function within the Azure Hub. I have only found Azure function tutorials on the HTTP triggers (there are lots so I know how to test that!) but I do not know the specifics of testing an Event hub trigger once deployed to Azure. So I started the function app within the Azure GUI and then started the streaming logs from VSC. I then right clicked on the function and selected to ‘Execute Function now’ within VSC with the test message (is this the correct way to test?):
200598-image.png

This resulted in the below logged entries with in live metrics:
200661-image.png

Nothing within the init.py is displayed and the request is below listed as an error:
200633-image.png

Run with AzureWebJobsStorage Value from local.settings.json:
This seemed better in that no exception errors occurred and a trace entry was recorded when I executed the function within VSC. However I get a 404 error:
200652-image.png
200653-image.png

This seems like an improvement. However:
1) I get the 404 error
2) I get the error messages within the dev environment (2nd screen grab); this is problem if I want to test and deploy.
3) My IoT Hub was receiving data throughout all of this and nothing was posted to the live metrics
4) How do I get to this view:
200671-image.png

Thanks for your help thus far; your comments on the above would be welcome.


image.png (63.3 KiB)
image.png (119.5 KiB)
image.png (188.9 KiB)
image.png (94.2 KiB)
image.png (70.7 KiB)
image.png (16.3 KiB)
image.png (51.4 KiB)
image.png (39.7 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ChrisU-9274 avatar image
0 Votes"
ChrisU-9274 answered

Thanks again for your comprehensive response. I have worked through you last response and have had good success today. See below:

• I opened the firewall ports as you suggested. I did not update the host.json as you suggested. Opening the ports seemed to work.

• Yes, I did use the VS code deploy button as your screen shot indicated and it did generate a .venv file:
201455-image.png

• I have double checked, and the The azure function configuration settings in the cloud match those in local.settings.json. Below shows where I got these values form.
201488-image.png

• Once I added the azure iot tools and toolkit my iot hub and devices were displayed within VSC without me doing anything:
201465-image.png

• I did notice that I had eventHubName configured incorrectly within the function.json file. I had it set to an event hub I have created, not the IoT Hub:
201500-image.png
I changed this to my IoT Hub and then redeployed the function App to Azure:
201542-image.png
This was likely the cause of the issues I was seeing (possibly not the firewall settings. I will test further to see)

• I deployed these updates to Azure and also managed to get the ‘function’ option to show my trigger when clicking on Functions, by clicking ‘Refresh’
201509-image.png
This allowed me to check that the code has been deployed correctly – see below the json file with the change shown above to prove this.
201489-image.png

TESTING.
So I ensured that the function was not running on the Azure platform at the time that this was tested locally. Good news! I did not get the error I had last time. This was with the new firewall settings and also the correct hub name detailed within the function.json file.

However when I ran the test you detailed I got 10 failed messages:
201533-image.png
However, as you can see above, at the same time I started receiving uplink data from the device.

I have now adjusted my code to read the data and assign specific values to variables. I have also added the decoder function to convert the environment data from hexadecimal to display the temperature and humidity data. See below and I have updated the repo here.
201519-image.png
I deployed the updated code to Azure and confirmed that this worked as below:
201525-image.png
I started the function with Azure and it is posting to the logs:
201543-image.png
201490-image.png

So this is good!

To surmise.
1) It is now working in that I am able to read the uplink data, sort through it and assign to variables.
2) I can now edit and test within the dev environment and then deploy successfully on the Azure platform.
3) Testing by sending D2C message to IoT Hub does not work. It would be good to have your feedback on this as to why you think this would be the case.


Next steps:
I now need to hook this up to my Azure SQL database and deposit the variable values within it. I do not quite understand defining the AzureWebJobStorage within the local.settings.json, as when I check the assigned storage with Azure nothing seems to have taken place there; if any activity has taken place here, where would you check?

Can you advise me and on any best practices in connecting to an Azure SQL DB, how to do it, or point me at resources that you think will help?

Finally. A massive thanks to you for getting me this far. I had almost given up numerous times!




image.png (66.4 KiB)
image.png (192.4 KiB)
image.png (33.9 KiB)
image.png (59.7 KiB)
image.png (63.2 KiB)
image.png (38.5 KiB)
image.png (41.4 KiB)
image.png (214.9 KiB)
image.png (407.4 KiB)
image.png (90.0 KiB)
image.png (88.2 KiB)
image.png (44.6 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

BrunoLucas-9843 avatar image
0 Votes"
BrunoLucas-9843 answered BrunoLucas-9843 edited

Hi @ChrisU-9274 ,

Glad to hear you've got it running.

Once you run it first time and see the messages flowing, you should also see this under the storage> blob>container

201585-image.png

It needs that to track the messages. Take another look and let me know. be careful also if you need to delete that or move resources around , the message will stay in the hub for the time you specify (I think default/minimum is one day). if you delete the storage that contains the message that was already sent and drop the storage data that tracks that, the hub my thrown an error)

Regarding querying Azure Sql Server, you will need something like this:
https://docs.microsoft.com/en-us/azure/azure-sql/database/connect-query-python?view=azuresql#create-code-to-query-your-database

You may also make sure your Sql Server allows for the connection:
https://docs.microsoft.com/en-us/azure/azure-sql/database/firewall-configure?view=azuresql#connections-from-inside-azure

if you start to have many azure resources you may also consider a vnet:
https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-overview

hope that helps


image.png (10.4 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.