question

OJB1-4839 avatar image
0 Votes"
OJB1-4839 asked SandervandeVelde42 commented

Azure IOT Hub Service Client SDK Issue --> Invoke Device Method fails after sixteen method calls

I am having issues with C# Microsoft.Azure.Devices Service SDK (current version 1.38.0) and experienced the same issue with former version1.77.1

I have a .NET 6 Core Web API project that is using the SDK to invoke device methods on the IOT Device (IOT Device is a seperate .NET 6 Worker Project)

Issue:

After invoking a device method exactly sixteen times in a row, the SDK returns an invalid error code 504101 within the try/catch block I'm using to handle the exception from the SDK. I have repeated the tests a number of times and perpelxed why it fails on the 17th time I invoke the method. The only way to resolve this is re-starting the project, then repeat the test and I then have the same issue.

Below is the Repository in my Web Api project that I use for calling the IOT Hub with the service client:

 public class CloudToDeviceRepo : ICloudToDeviceRepo
 {
     private readonly IConfiguration _config;
     private readonly ILogExtension _logExtension;
     private static ServiceClient? _serviceClient;
     private readonly string _primaryConnectionString;
    
     private static string? LogData { get; set; }
    
     public CloudToDeviceRepo(
     IConfiguration config,
     ILogExtension logExtension)
     {
         _config = config;
         _logExtension = logExtension;
         _primaryConnectionString = _config.GetSection("AzureIotHub").GetValue<string>("PrimaryConnectionString");
         _serviceClient = ServiceClient.CreateFromConnectionString(_primaryConnectionString);
     }
    
     /// <summary>
     /// Example methodName = "RestartIotDevice"
     /// </summary>
     /// <param name="deviceId"></param>
     /// <param name="methodName"></param>
     /// <param name="methodPayload"></param>
     /// <param name="responseTimeoutSeconds"></param>
     /// <returns></returns>
     public async Task<CloudToDeviceMethodResult> InvokeDeviceMethod(string deviceId, string methodName, string methodPayload, int responseTimeoutSeconds)
     {
         var methodInvocation = new CloudToDeviceMethod(methodName)
         {
             ResponseTimeout = TimeSpan.FromSeconds(responseTimeoutSeconds),
         };
         methodInvocation.SetPayloadJson(methodPayload);
    
         try
         {
              var response = await _serviceClient!.InvokeDeviceMethodAsync(deviceId, methodInvocation);
              return response;
          }
          catch (Exception ex)
          {
              LogData = "IotHubService-API encountered an exception when invoking a cloud to device method with method name [" + methodName + "] for an iot device with id [" + deviceId + "]. Azure IOT Hub returned an error with message: [" + ex.Message + "]";
              await _logExtension.WriteLogEvent<CloudToDeviceRepo>("Error", "IOT Devices", "Invoke Device Method", deviceId, LogData, null!);
             throw;
         }
     }
 }

In my program startup, I run the repository as a singleton service (and had the same issue using transient or scoped)

 // Service for calling Azure IOT Hub
 builder.Services.AddSingleton<ICloudToDeviceRepo, CloudToDeviceRepo>();

Below is a sample method handler in my IOT Device Project. I am getting the same issue with the service client no matter what device method I call on the IOT Device from the Hub, using the Service Client SDK.

 private async Task<MethodResponse> TestMethod(MethodRequest methodRequest, object userContext)
 {
     // Acknowlege the direct method call with a 200 success message
     string result = $"{<!-- -->{\"result\":\"Executed direct method: {methodRequest.Name}\"}}";
     //return Task.FromResult(new MethodResponse(Encoding.UTF8.GetBytes(result), 200));
     return new MethodResponse(Encoding.UTF8.GetBytes(result), 200);
 }

Below is the details of the exception I receive in my Web API after invoking the device method after sixteen times:

228828-exception-screenshot.jpg


When inpecting the exception, it seems to be that the issue resides from something within the SDK, related to the HttpClientHelper:

Exception Message:

{"{\"Message\":\"{\\\"errorCode\\\":504101,\\\"message\\\":\\\"Timed out waiting for the response from device.\\\",\\\"trackingId\\\":\\\"EFEA228A0E5F4E96932DB46DCB36D9B8-G2:-TimeStamp:2022-08-07T10:32:51.149877054+00:00\\\",\\\"timestampUtc\\\":\\\"2022-08-07T10:32:51.149877054+00:00\\\",\\\"info\\\":{\\\"timeout\\\":\\\"00:00:30\\\"}}\",\"ExceptionMessage\":\"\"}"}

ex.StackTrace

at Microsoft.Azure.Devices.HttpClientHelper.<ExecuteAsync>d_36.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Devices.HttpClientHelper.<ExecuteWithCustomOperationTimeoutAsync>d
33.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Devices.HttpClientHelper.<PostAsync>d
26`2.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at IotHubService_API.Repositories.AzureIotHub.CloudToDeviceRepo.<InvokeDeviceMethod>d
_9.MoveNext() in G:\My Drive\MyProject\Repositories\AzureIotHub\CloudToDeviceRepo.cs:line 48

ex.SerializationStackTraceString

at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func`3 modifyRequestMessageAsync, Func`2 isMappedToException, Func`3 processResponseMessageAsync, IDictionary`2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteWithCustomOperationTimeoutAsync(HttpMethod httpMethod, Uri requestUri, TimeSpan operationTimeout, Func`3 modifyRequestMessageAsync, Func`2 isMappedToException, Func`3 processResponseMessageAsync, IDictionary`2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T,T2](Uri requestUri, T entity, TimeSpan operationTimeout, IDictionary`2 errorMappingOverrides, IDictionary`2 customHeaders, CancellationToken cancellationToken)
at IotHubService_API.Repositories.AzureIotHub.CloudToDeviceRepo.InvokeDeviceMethod(String deviceId, String methodName, String methodPayload, Int32 responseTimeoutSeconds) in G:\My Drive\MyProject\Repositories\AzureIotHub\CloudToDeviceRepo.cs:line 48

I really dont know to how to troubleshoot this further, I had checked the diagnositics health reports in the Azure portal for the IOT Hub account, but no issues were reported. I do beleive the issue lies somewhere at my end i.e. calling the Azure Hub using the Service Client SDK from my Web Api project. The fact that I can suscessfully invoke a device method the first sixteen times without issue would have me beleive there is nothing wrong with my own code implemtation, but is very stange how the SDK returns the exception on the 17th run each time before I restart the project. I'm wandering whether the SDK is suffering from some form of socket exhaustion and somehow not handling the HttpClient connection correctly.

Update 1 --> In order to elimiante anything on the client side in my Web Api project when calling the Service Hub client, I used a sample template from the IOT Hubs samples.

I am still getting the same issue after invoking an IOT device method after calling it more than sixteen times in a row, leaving a two second gap in between each call.

Console Application code sample below:

 using System;
 using System.Threading.Tasks;
 using Microsoft.Azure.Devices;
    
 namespace IotHubServiceClientTest
 {
     /// <summary>
     /// This sample illustrates the very basics of a service app invoking a method on a device.
     /// </summary>
     internal class Program
     {
         private static ServiceClient s_serviceClient;
    
         private static async Task Main(string[] args)
         {
             Console.WriteLine("IoT Hub Quickstarts - InvokeDeviceMethod application.");
    
             // Create a ServiceClient to communicate with service-facing endpoint on your hub.
             s_serviceClient = ServiceClient.CreateFromConnectionString("HostName=myservicehubconnectionstring");
    
             for (int i = 1; i < 18; i++)
             {
                 Console.WriteLine(i);
                 await InvokeMethodAsync("F4-7B-09-A1-13-64");
                 await Task.Delay(2000); // Wait 2 secs before next call.
             }
    
             s_serviceClient.Dispose();
    
             //Console.WriteLine("Press Enter to exit.");
             Console.ReadLine();
         }
    
         // Invoke the direct method on the device, passing the payload.
         private static async Task InvokeMethodAsync(string deviceId)
         {
             var methodInvocation = new CloudToDeviceMethod("SetTelemetryInterval")
             {
                 ResponseTimeout = TimeSpan.FromSeconds(30),
             };
             methodInvocation.SetPayloadJson("10");
    
             Console.WriteLine($"Invoking direct method for device: {deviceId}");
    
             try
             {
                 // Invoke the direct method asynchronously and get the response from the simulated device.
                 var response = await s_serviceClient.InvokeDeviceMethodAsync(deviceId, methodInvocation);
    
                 Console.WriteLine("");
                 Console.WriteLine($"Response status: {response.Status}, payload:\n\t{response.GetPayloadAsJson()}");
             }
             catch (Exception ex)
             {
                 Console.WriteLine("");
                 Console.WriteLine($"Exception message: {ex.Message}");
                 Console.WriteLine("");
                 Console.WriteLine($"Exception inner exception: {ex.InnerException}");
                 Console.WriteLine("");
                 Console.WriteLine($"Exception stack trace: {ex.StackTrace}");
             }
         }
     }
 }

Below is a screenshot showing the output error given from the service client:

228873-consoleclienttestexception.jpg


I will keep troublshooting, I guess my next step will be to use a sample app code for the IOT device end in order to receive the request invoking the device method. Weird thing is that no errors or exceptions are noted in my IOT evice app, all that happens is on the 17th time I invoke a device method, the device method handler simply dosen't run at all which Is noteably why I get a timeout exception returned from the service client sdk.

Update 2 --> I have now implemented a test IOT Device client, again using an IOT Hubs code sample. I am still getting the issue so can now confirm that using the most basic example of invoking an IOT Device method fails after the first sixteen iterations.

Below is the code sample for the IOT Device Client:

 namespace IotDeviceClientTest
 {
     internal class Program
     {
         private static DeviceClient s_deviceClient;
         private static readonly TransportType s_transportType = TransportType.Mqtt;
    
         private static async Task Main(string[] args)
         {
             Console.WriteLine("IoT Hub Quickstarts #1 - Simulated device.");
    
             // Connect to the IoT hub using the MQTT protocol
             s_deviceClient = DeviceClient.CreateFromConnectionString("HostName=myIotHubDeviceClientConnectionString", s_transportType);
    
             // Create a handler for the direct method call
             await s_deviceClient.SetMethodHandlerAsync("SetTelemetryInterval", SetTelemetryInterval, null);
    
             Console.ReadLine();
         }
    
         private static Task<MethodResponse> SetTelemetryInterval(MethodRequest methodRequest, object userContext)
         {
             var data = Encoding.UTF8.GetString(methodRequest.Data);
    
             // Acknowlege the direct method call with a 200 success message.
             string result = $"{<!-- -->{\"result\":\"Executed direct method: {methodRequest.Name}\"}}";
    
             Console.WriteLine("");
             Console.WriteLine(result);
                
             return Task.FromResult(new MethodResponse(Encoding.UTF8.GetBytes(result), 200));
         }
     }
 }

Update 3 --> My next test was to invoke the device method directly from the Azure IOT Hub portal, again I get the same issue so we can now rule out the IOT Hub Service Client SDK. As above, invoking the device meothod fails after 16qty times.

Screenshot shows the portal is waiting once it's sent the 17th method invoke request:

228892-invokedevicemethodfromazureportal.jpg

Screenshot shows the error message on the 17th method invoke:

228875-invokedevicemethodfromazureportal2.jpg

Other troubleshooting steps I've tried:

Tried registering a new IOT Device in the Azure portal, and tested invoking methods directly from the hub for that device.
Changed the Device Id to a simple name "TestDevice1" in case the SDK didnt like the fact I was using a MAC address as the id.
Tried experiementing with using various different Reponse Timeout periods
I fired the first 16qty device method invokes, then waited a few mins before firing off the 17th method invoke, the 17th call still failed.

My overall conclusion so far as follows:

  • There is either an issue with Azure IOT Hubs for my particular account

  • Or there is an issue with the Azure IOT Hubs C# Device Client SDK (Microsoft.Azure.Devices.Client current ver.1.41.1 but I have tried previous versons of the SDK as well, going down as far as 1.30.0)

  • Or there is some limitation I'm not aware of in being able to invoke device client messages more than 16qty times in a row, in a fairly short duration of 2 seconds apart from each call (seems unlikely going by the MS throttling quotas which are well within the tolerences of my basic testing)

azure-iot-hubdotnet-aspnet-core-webapiazure-iot-sdkdotnet-iot
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

SandervandeVelde42 avatar image
0 Votes"
SandervandeVelde42 answered

Hello @OJB1-4839,

thanks for reaching out on MS Learn Q&A.

This is a community-driven forum (although moderation is done together with Microsoft).

Because you point out a potential issue with this SDK, I recommend creating an Issue on https://github.com/Azure/azure-iot-sdk-csharp/issues so the team behind this SDK can check it out.

· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Sander,

I now have posted the topic as per your recommendation, link to the issue reported --> https://github.com/Azure/azure-iot-sdk-csharp/issues/2557
I have cut down some of the notes on the ticket considering that much of the above info was following through some of the earlier troubleshooting whilst I was trying to narrow down the route cause.

Ultimately, I still find the same issue when invoking the device methods directly from the Azure Portal, and using a very basic code sample for the IOT Device that receives and responds to the device method invoke for just 16ty times in a row, before it then fails with a timeout reported in Azure.



I may be waiting a while for someone to investigate this issue through GitHb, could you please at least confirm whether it should be possible to invoke device methods more than 16qty times in a row? This would determine how much further I go in my own troubleshooting, thanks

1 Vote 1 ·

Hello @OJB1-4839,

I tested direct methods calls with this simple client:

 namespace FailingDirectMethodDevice
 {
     internal class Program
     {
         static void Main(string[] args)
         {
             Console.WriteLine("Waiting for methods!");
    
             var cs = "[connectionstring]";
    
             var client = DeviceClient.CreateFromConnectionString(cs);
    
             client.SetMethodDefaultHandlerAsync(HandleDefaultMethods, client).ConfigureAwait(false);
    
    
             Console.WriteLine("Press a key to exit");
    
             Console.ReadKey();
         }
    
         private static int index = 0;
    
         private static Task<MethodResponse> HandleDefaultMethods(MethodRequest methodRequest, object userContext)
         {
             index++;
    
             Console.WriteLine($"Default method  #{index}: {methodRequest.Name} handled with body {methodRequest.DataAsJson}");
    
             return Task.FromResult(new MethodResponse(new byte[0], 200));
         }
     }
 }

I was able to make more than 16 direct method calls:

229268-image.png

I then tested it with your client code example.

I had to fix this one line because it did not compile:

 //// NEW
 string result = "{\"result\":\"Executed direct method: methodRequest.Name\"}";

This also worked well for me.


1 Vote 1 ·
image.png (130.0 KiB)

Can you confirm the response message is well-formed, every time?

Next, a direct method expects a device to be connected and respond within the timeout constraints.

Can you double check if your client is still connected?

0 Votes 0 ·
OJB1-4839 avatar image OJB1-4839 SandervandeVelde42 ·

Hi sander, which version of .NET did you use in your test and which version of the Client SDK did you use?

I'm still finding the same issue currently, thanks

0 Votes 0 ·
Show more comments
OJB1-4839 avatar image
1 Vote"
OJB1-4839 answered SandervandeVelde42 commented

The issue was caused by including the transport type in the connection string which I had set to MQTT.

So by not including the transport type, the connection resolves to AMQP by default and this is now working as expected.

I since found something on the MS Docs that says "Direct methods are HTTPS-only from the cloud side and MQTT, AMQP, MQTT over WebSockets, or AMQP over WebSockets from the device side." https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-direct-methods

I'm absolutely gutted I missed this point and realize now why it wasnt working. My apologies to the support team and massive thank you to Sander for pointing me in the right direction, I will mark this as resolved, many thanks


· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @OJB1-4839 ,

I just noticed the GitHub issue is being closed and a fix will be provided by the team.

Good job identifying this issue and finding the work-around.


0 Votes 0 ·