What could cause this error: Cluster creation failed due to an unexpected system error from a runtime related dependency

Question

During one of the scheduled executions of a pipeline, an error occurred with the following details:

Error code: [5000](https://go.microsoft.com/fwlink/?linkid=2117168#error-code-5000)
Failure type: User configuration issue
Details: Cluster creation failed due to an unexpected system error from a runtime related dependency.

The output of the Dataflow shows the following:

"ErrorInfo": [{ "Message": "Cluster creation failed due to an unexpected system error from a runtime related dependency.", "ErrorCode": "CLUSTER_LIVY_URL_NOT_FOUND" }]

What could be the possible causes of this error?

Answer 1

Hi @Veena

Thanks for the question and using MS Q&A platform.

Based on the error message "Cluster creation failed due to an unexpected system error from a runtime related dependency" with error code "CLUSTER_LIVY_URL_NOT_FOUND" indicates that there was an issue with the Livy service in the cluster. Livy is a RESTful web service for Apache Spark that enables easy submission of Spark jobs from anywhere.

The possible causes of this error could be:

1. The Livy service is not running or is not configured correctly.
2. There is a network issue preventing the pipeline from connecting to the Livy service.
3. There is an issue with the Spark runtime or dependencies.

To resolve this issue, you can try the following steps:

1. Check the status of the Livy service in the cluster and ensure that it is running.
2. Check the network connectivity between the pipeline and the Livy service.
3. Check the Spark runtime and dependencies to ensure that they are configured correctly.

If the issue persists, you can try restarting the Livy service or the entire cluster. You can also check the Azure portal for any service health issues or outages that may be affecting the cluster. If the issue still persists, you can contact Microsoft support for further assistance.

I hope this helps. Please let me know if you have any further questions.