Ingest data from IoT Hub to data explorer

Question

I have some divices connected to IoT Hub thru Node Red. Then I want to ingest the data from these devices to data explorer. I have created the cluster, database anbd the connection. When I try to connect I get the following error applying the command ".show ingestion failures"

"OperationId": 0c25117e-cb0d-403f-b34f-c2c8e18fec7b,
"Database": MiBaseDatos,
"Table": Table1,
"FailedOn": 2022-11-04T10:45:42.3883537Z,
"IngestionSourcePath": MULTIPLE_SOURCES. Examples: https://75wkstrlddatosdb01.blob.core.windows.net/20221104-readyforaggregation/eh-iothub-ehub-borja-hub-22410154-37dbb7b14f-partition-0-seq-375-375-count-1-qkn.multijson.gz,https://xdakstrlddatosdb00.blob.core.windows.net/20221104-readyforaggregation/eh-iothub-ehub-borja-hub-22410154-37dbb7b14f-partition-0-seq-376-378-count-3-g0s.multijson.gz,
"Details": BadRequest_InvalidBlob: The parameter is incorrect. (os error -2147024809),
"FailureKind": Permanent,
"RootActivityId": 9f5b0bd6-2459-404e-b0e0-931a0e6ddb66,
"OperationKind": DataIngestPull,
"OriginatesFromUpdatePolicy": false,
"ErrorCode": BadRequest_InvalidBlob,
"Principal": dstsapp=KustoIngestionProd,
"ShouldRetry": false,
"User": ,
"IngestionProperties": [Format=MultiJson/mandatory, IngestionMapping=[{"column":"Device","datatype":"","Path":"$.Device"},{"column":"Data","datatype":"","Path":"$.Data"},{"column":"IotHubDeviceId","datatype":"","Path":"$.iothub-connection-device-id"},{"column":"IotHubEnqueuedTime","datatype":"","Path":"$.iothub-enqueuedtime"}], ValidationPolicy=[Options=ValidateCsvInputConstantColumns, Implications=BestEffort, IsDetailedErrorReportingEnabled=False], Tags=[ToStringEmpty], IngestIfNotExists=[ToStringEmpty], ZipPattern=[null]],
"NumberOfSources": 2

It looks that the format is not correct, but I do not understanb why and what i have to change.
The format I am using is json.

Answer 1

Hello @borja humet mir ,

testing the ingestion of messages sent by IoT devices can be done using the IoT Explorer.

There are two things to take into account:

1. The UI can only cope with messages in plain JSON format
2. try to avoid the $default consumer group

First, make sure you are sending messages in the actual JSON format. Perhaps you can add an extra test device (eg. written in C#) or use this simulation and start sending proper JSON.

The IoT Hub default endpoint (used by the IoT Explorer) acts like an Event Hub so you need to define consumer groups for separate consumers (eg. the IoT Explorer). If not, these consumers will 'fight' for both receiving the same message instead of each getting a copy.

Answer 2

Hello,

Adding few more troubleshooting steps..!

Connection

First we can check that the connection between IoT Hub and Data Explorer is set up correctly.

The format or contents of the files being ingested into Data Explorer.

The error message specifically mentions that the parameter is incorrect and there is an OS error. Make sure that the data being ingested matches the format specified in the ingestion properties and the mapping is correct.

Source Paths

The error message mentions multiple source paths, so make sure that all of the paths exist and the files are accessible.

You can try validating the data using a JSON validator or by checking the structure of the data against the expected format.

If none of these steps solve the issue, you may need to contact Microsoft Azure support for further assistance or add a comment in the below section and we are happy to help you in this matter.