Ingest data from IoT Hub to data explorer

borja humet mir 1 Reputation point
2022-11-04T11:53:48.98+00:00

I have some divices connected to IoT Hub thru Node Red. Then I want to ingest the data from these devices to data explorer. I have created the cluster, database anbd the connection. When I try to connect I get the following error applying the command ".show ingestion failures"

"OperationId": 0c25117e-cb0d-403f-b34f-c2c8e18fec7b,
"Database": MiBaseDatos,
"Table": Table1,
"FailedOn": 2022-11-04T10:45:42.3883537Z,
"IngestionSourcePath": MULTIPLE_SOURCES. Examples: https://75wkstrlddatosdb01.blob.core.windows.net/20221104-readyforaggregation/eh-iothub-ehub-borja-hub-22410154-37dbb7b14f-partition-0-seq-375-375-count-1-qkn.multijson.gz,https://xdakstrlddatosdb00.blob.core.windows.net/20221104-readyforaggregation/eh-iothub-ehub-borja-hub-22410154-37dbb7b14f-partition-0-seq-376-378-count-3-g0s.multijson.gz,
"Details": BadRequest_InvalidBlob: The parameter is incorrect. (os error -2147024809),
"FailureKind": Permanent,
"RootActivityId": 9f5b0bd6-2459-404e-b0e0-931a0e6ddb66,
"OperationKind": DataIngestPull,
"OriginatesFromUpdatePolicy": false,
"ErrorCode": BadRequest_InvalidBlob,
"Principal": dstsapp=KustoIngestionProd,
"ShouldRetry": false,
"User": ,
"IngestionProperties": [Format=MultiJson/mandatory, IngestionMapping=[{"column":"Device","datatype":"","Path":"$.Device"},{"column":"Data","datatype":"","Path":"$.Data"},{"column":"IotHubDeviceId","datatype":"","Path":"$.iothub-connection-device-id"},{"column":"IotHubEnqueuedTime","datatype":"","Path":"$.iothub-enqueuedtime"}], ValidationPolicy=[Options=ValidateCsvInputConstantColumns, Implications=BestEffort, IsDetailedErrorReportingEnabled=False], Tags=[ToStringEmpty], IngestIfNotExists=[ToStringEmpty], ZipPattern=[null]],
"NumberOfSources": 2

It looks that the format is not correct, but I do not understanb why and what i have to change.
The format I am using is json.

Azure IoT Hub
Azure IoT Hub
An Azure service that enables bidirectional communication between internet of things (IoT) devices and applications.
1,157 questions
Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
508 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Sander van de Velde | MVP 31,211 Reputation points MVP
    2022-11-06T22:40:31.127+00:00

    Hello @borja humet mir ,

    testing the ingestion of messages sent by IoT devices can be done using the IoT Explorer.

    There are two things to take into account:

    1. The UI can only cope with messages in plain JSON format
    2. try to avoid the $default consumer group

    First, make sure you are sending messages in the actual JSON format. Perhaps you can add an extra test device (eg. written in C#) or use this simulation and start sending proper JSON.

    The IoT Hub default endpoint (used by the IoT Explorer) acts like an Event Hub so you need to define consumer groups for separate consumers (eg. the IoT Explorer). If not, these consumers will 'fight' for both receiving the same message instead of each getting a copy.

    0 comments No comments

  2. QuantumCache 20,261 Reputation points
    2023-02-27T06:34:02.7433333+00:00

    Hello,

    Adding few more troubleshooting steps..!

    Connection

    First we can check that the connection between IoT Hub and Data Explorer is set up correctly.

    The format or contents of the files being ingested into Data Explorer.

    The error message specifically mentions that the parameter is incorrect and there is an OS error. Make sure that the data being ingested matches the format specified in the ingestion properties and the mapping is correct.

    Source Paths

    The error message mentions multiple source paths, so make sure that all of the paths exist and the files are accessible.

    You can try validating the data using a JSON validator or by checking the structure of the data against the expected format.

    If none of these steps solve the issue, you may need to contact Microsoft Azure support for further assistance or add a comment in the below section and we are happy to help you in this matter.

    0 comments No comments