Optimizing Azure Synapse Analytics for Spark Pool and Mapping Data Flows

Question

I am encountering errors involving Livy sessions when running multiple pipelines in Synapse Notebook and mapping data flows. The errors are as follows:

For notebooks:

• Operation on target forecast_constrained failed: Exception: Failed to create Livy session for executing notebook. LivySessionId: 1392Notebook: raw_forecast_constrained_outbound. --> : Livy session has failed. Session state: Error. Session yields Failed result.
• Operation on target data_cleaning_saleshistory failed: Exception: Failed to create Livy session for executing notebook. LivySessionId: 219Notebook: data_cleaning_saleshistory. --> CLUSTER_CREATION_TIMED_OUT: Livy session has failed. Session state: Error. Error code: CLUSTER_CREATION_TIMED_OUT. [plugins.azx-asa-mspbi-dp-p-01.azxaspd365dpp1.219 WorkspaceType: CCID:<90b4f648-6a6f-4520-8943-a0161a9d9874>] The job has surpassed the max number of seconds it can be in ResourceAcquisition state [2000s], likely due to the cluster creation timing out, so ending the job. Source: Dependency.

For mapping data flows:

• Hit unexpected exception, please retry. If the problem persists, please contact Azure Support"_
• `{ "errorCode": "BadRequest", "message": "Operation on target data_quality_outbound failed: [traceId=plugins.AZX-ASA-MSPBI-DP-P-01.General-8-1-7f9e6b13-5d0a-4061-b875-3e2cc54d3abb.3367d738-cbfb-4bc4-af04-8119bb70666f WorkspaceType:

Answer 1

@Aditya Singh
Thank you for reaching out to the community forum with your query.

We apologize for the inconvenience caused by the recent outage. We want to inform you that an outage was reported in the Europe region, which may have impacted your notebooks. However, the issue has now been resolved, and the service has been restored.

This can happen due to various reasons like network issues, credential failures, or service outages. Azure Data Factory has an internal recovery process that monitors all the runs and starts them when it notices something went wrong.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.