How to debug Azure IoT Operation Data pipeline failure

Hennessy, Tim 25

I am working through an Azure IoT Operations QuickStart article regarding the Data processor. After setting up the passthrough pipeline and saving it, it never syncs correctly.

I dumped the pipeline information and get this error information

Provisioning Status: Error: Code: RunnerGRPC Message: The operator failed to establish a connection with the Runner-worker. Failure Cause: Unknown Operation Id: c2048032-b4d2-4313-98b0-3a0601313f74*6DD9C1D5D399783A83E02115E1656360F5FC51CBFD7B58D5966752849536D803 Status: Provisioning

So, I decided to delete the pipeline and start over. But I'm unable to delete the pipeline, the pipeline shows this status

Provisioning Status: Error: Operation Id: 9350cec9-f52b-44a3-8576-6ca33f827db7*EFE21F6EAB7808A422435DFB431ECCD15EBA994358893FA59145E702BAD8DAFA Status: Deleting

Is this a known issue with the QuickStart? What steps can I take to get the pipeline sync to succeed?

Thanks,

LeelaRajeshSayana-MSFT 15,886 Reputation points Microsoft Employee

2024-03-26T23:05:47.69+00:00
Hi @Hennessy, Tim Greetings! Welcome to Microsoft Q&A forum. Thank you for posting this question here.

The first message indicates that the resource is still in provisioning state when you tried to delete it which could likely caused the error.

To get more information on the status of the pipeline, try running the below pipeline

kubectl describe pipelines passthrough-data-pipeline -n azure-iot-operations

Please refer the troubleshooting guide section Find pipeline error codes for more information on this.

Kindly keep us posted on your findings.
Hennessy, Tim 25 Reputation points

2024-03-27T00:00:03.63+00:00

Hi @LeelaRajeshSayana-MSFT thanks for the response. The error information I provided in the original post is from the command you mentioned. After not being able to delete the pipeline, I repeated the steps and got the same sync failure with the error information Message: The operator failed to establish a connection with the Runner-worker. And you can see the status is Provisioning. If you can provide any insight on restarting modules, or additional logs, I'd appreciate it.
LeelaRajeshSayana-MSFT 15,886 Reputation points Microsoft Employee

2024-03-27T20:59:40.84+00:00

Hi @Hennessy, Tim Thank you for your response. I brought this issue to our PG attention. I will let you know as soon as we get more findings from the team on this.

1 answer

LeelaRajeshSayana-MSFT 15,886 Reputation points Microsoft Employee

2024-03-27T22:44:58.9266667+00:00

Hi @Hennessy, Tim The product team's initial analysis on this issue, based on the logging message, seems to be a deployment issue. The team has requested for additional logs which can be collected using the CLI command az iot ops support create-bundle. The command creates a standard support bundle zip which you can share with us offline to further investigate this issue.

Once you generated the support bundle, please send an email to azcommunity@microsoft.com with the below details, so that we can work closely on this matter.

Update

Root cause is still being analyzed for the behavior. Thread will be updating with further findings. As a work around, create a new set of resources to overcome the issue.

If the response helped, please do click Accept Answer and Yes for the answer provided. Doing so would help other community members with similar issue identify the solution. I highly appreciate your contribution to the community.
Please sign in to rate this answer.
Hennessy, Tim 25 Reputation points

2024-03-29T17:47:07.8733333+00:00

An update: I zipped up the support bundle and sent that off. Then, for unrelated reasons, had to rebuild my Codespace environment which trashed my setup. So, I decided to start from scratch and went through each step. When I got to the sync failed step it worked. I don't know why. The logs might hold the information but I no longer have that AIO cluster setup. Please let me know if you find something. I appreciate the support.

LeelaRajeshSayana-MSFT 15,886 Reputation points Microsoft Employee

2024-03-29T20:22:40.6+00:00

Hi @Hennessy, Tim Thank you for connecting with us offline and sharing the required logs. Our product team is working on analyzing the logs for further insights. I will keep you posted once we are done with the RCA.

Glad to know that a fresh set up has helped you overcome this issue. If you are content with the engagement and support received on this thread and I appreciate if you can leave a Yes rating on thread.

Hennessy, Tim 25 Reputation points

2024-04-02T19:21:32.24+00:00

Hi. Quick recap. Rebuilding AIO and codespace from scratch solved this problem without a root cause. However, after saving pipelines and sync'ing them, the very last pipeline I was working on experienced the KV secret issue. After updating the pipeline and saving it with those values, I'm back to this issue where it says it saved but not syncrhonized. Or it shows it sync'ing endlessly as shown here

I tried adding a second version which did the same thing as the first. I also pulled the status of the pipelines and the error is the same as the very first time I experienced the problem.

Provisioning Status: Error: Code: RunnerConnectionFailure Message: {"code":"RunnerConnectionFailure","message":"Failed to establish a connection with the runner-worker. Ensure runner-worker is healthy and try again.","innererror":{"code":"Unknown","message":"context deadline exceeded","error":{}}}

Failure Cause: Unknown

Operation Id: 06affee3-9abe-445f-acfd-0152b6a726e1*8288E119752B84861E7DB9417D1E031E69A3DC61665ED36C097DDE8C8D6DB99D

Status: Updating

Events: <none>

I have a new set of log bundles I can send you. I'll do that using the same email as before. Thanks.

Hennessy, Tim 25 Reputation points

2024-04-16T17:21:04.75+00:00

Still hoping to get information on the pipeline error above. The logs have been sent, just waiting for additional information.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to debug Azure IoT Operation Data pipeline failure

1 answer

Your answer