Spark cluster not found error in ADF

Rahul Reddy Kankanala 0 Reputation points
2023-03-01T10:32:27.4233333+00:00

While running ADF pipelines we are getting the below error.

"Spark cluster not found" error in Azure Data Factory.

Increased the timeout of debug session to 3 hours, still no use

User's image

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,539 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. BhargavaGunnam-MSFT 26,136 Reputation points Microsoft Employee
    2023-03-07T21:17:34.6333333+00:00

    Hello Rahul Reddy Kankanala,

    Welcome to the MS Q&A platform.

    Are you seeing this error on the debug pipelines or trigger pipelines?

    If you see the error with Dataflow debug, the cluster has TTL, and if nothing is going on for a while, then the cluster will die.

    For example: if copy activities are running before Dataflow and taking more time than TTL then you might see this error.

    Please let me know if you have any further questions,

    0 comments