Azure Data Synapse pipeline failing due to "Spark cluster not found"

Question

Intermittent failure on long running dataflow jobs: Operation on target copySpecifiedTableToDirectoryLocation failed: Spark cluster not found The pipeline succeeds about half the time and fails the other half. There doesn't seem to be any rhyme or reason to it. No resource errors (OOM, bash fail, worker fail) and no Spark monitoring dashboard to check myself on integrated runtime. Is there a more performant, more controlled way to run spark from Synapse? Maybe a dedicated cluster?

Answer 1

Hi Vincent Brandon,

Thank you for posting query in Microsoft Q&A Platform.

At this moment we don't have something called Dedicated Spark Pools. We only have Spark Pools, which will spin up compute resources during execution. You are issue may be transient issue. Kindly re-try and see if it helps.

Sometimes if any outage in service that may also result in this kind of transient issues. Kindly review Azure Status page also.

I forwarded your feedback of having dedicated Spark Pools to Internal team attention. I would encourage you as well to have a feedback item created for same using below link. https://feedback.azure.com/d365community/forum/9b9ba8e4-0825-ec11-b6e6-000d3a4f07b8

Internal team monitors feedbacks there and consider them for future implementations.

Hope this helps. Please let me know if any further queries.

---

Please consider hitting Accept Answer button. Accepted answers help community as well.