Getting runtime state is either null or still InProgress error from Spark in Azure Data flow.

Pawan Bisht 1 Reputation point
2022-12-06T13:40:56.307+00:00

I was working on Azure data flows.
While merging two parquet file I am getting the error : 

   The spark job of Dataflow completed, but the runtime state is either null or still InProgress.

It was successfully reading the data but not able to close the pipeline after running for approx. an hour. And abruptly failed the pipeline with the above msg.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,534 questions