Getting error like 'FAILED CLUSTER CREATION' for apache spark pool compute cluster

Question

Apache spark pool compute cluster is not running since 20th July. error show sin 'LivySessionCreationTimeoutException: Livy session creation timed out.' has anybody else faced this issue? I have created a new cluster and tried but that's showing the same issue.

Answer 1

Hi @Ghosh, Paramita

Thanks for the question and using MS Q&A platform.

LivySessionCreationTimeoutException: Livy Session Creation Timed Out This error typically occurs when the Livy session creation process takes longer than the specified timeout period. Livy is a REST interface for interacting with Apache Spark.

Here are some troubleshooting steps that might help you.

Check Cluster Resource Utilization: Verify that the cluster has sufficient resources (e.g., CPU, memory, and nodes) to handle the workload. You can check the cluster's resource utilization metrics in Azure Synapse Analytics.

Verify Network Connectivity: Ensure that there are no network connectivity issues between the client and the Spark cluster. Check the Azure Synapse Analytics network settings and firewall rules.

Review Spark Cluster Configuration: Double-check the Spark cluster configuration, including the Spark version, node size, and resource allocation. Ensure that the configuration is correct and suitable for your workload.

Livy Configuration: Review the Livy configuration settings, such as timeouts and concurrency settings. Adjust these settings if necessary to accommodate your workload.

Try a Different Node Size: If you're using a small node size, try increasing the node size to see if it resolves the issue.

I hope this information helps, please do let us know if you have any further queries.