Azure Synapse Spark Session stops ubruptly while processing and throwsLivyHttpRequestFailure 500 error

Question

Hi team,

When we try to run a synapse notebook written in python language, getting LivyHttpRequestFailure  500 error and stops abruptly from execution. Here is the error message for reference. This is consistently reproducible , happens every time.

{
	"status": {
		"Status": 2,
		"ExecutionStatusCode": null,
		"Output": {
			"status": {
				"id": "2805",
				"poolId": null,
				"sessionAcquiredFromPool": null,
				"clusterAcquiredFromPool": null,
				"name": "roster1_processor_idr1synspark_1699626579",
				"sparkPoolName": "idr1synspark",
				"workspaceName": "idr1-synapse-centralus-dev",
				"appId": "application_1699614407948_0173",
				"submitterId": "2864980c-2195-47d6-bc42-d8b4ff8e3e11",
				"errorInfo": [],
				"owner": null,
				"proxyUser": null,
				"kind": "spark",
				"isNewLivySession": false,
				"log": [
					"\t queue: default",
					"\t start time: 1699626589516",
					"\t final status: UNDEFINED",
					"\t tracking URL: http://vm-22b70064:8088/proxy/application_1699614407948_0173/",
					"\t user: trusted-service-user",
					"23/11/10 14:29:49 INFO ShutdownHookManager: Shutdown hook called",
					"23/11/10 14:29:49 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-88308f89-8cec-45ec-968f-a772900330c6",
					"23/11/10 14:29:49 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-0a5bf0a0-093b-4c30-89f8-2f701e4d717d",
					"\nstderr: ",
					"\nYARN Diagnostics: "
				],
				"state": "busy",
				"result": "uncertain",
				"appInfo": {
					"sparkUiUrl": "..............",
					"impulseErrorCode": null,
					"isStreamingQueryExists": "false",
					"impulseClassification": null,
					"impulseTsg": null,
					"isSessionTimedOut": "false",
					"driverLogUrl": "....."
				},
				"livyInfo": {
					"jobCreationRequest": {
						"jars": [],
						"pyFiles": [],
						"files": [],
						"driverMemory": "28g",
						"driverCores": 4,
						"executorMemory": "28g",
						"executorCores": 4,
						"numExecutors": 1,
						"archives": [],
						"name": "roster_queue_processor_idrsynspark_1699626579",
						"conf": {
							"livy.server.session.timeout": "4h",
							"admin.currentOperation.livy.server.session.timeout": "replace",
							"spark.dynamicAllocation.enabled": "false",
							"spark.dynamicAllocation.minExecutors": "1",
							"spark.dynamicAllocation.maxExecutors": "2",
							"spark.autotune.trackingId": "0dc81d9f-013f-4758-8972-0b2cc9a2cf05",
							"spark.synapse.context.pipelinejobid": "255ae8fa-b6b2-4278-bec3-5c5d843a25c7",
							"spark.synapse.context.activityrunid": "e829800c-7916-402a-97a6-034eaf61a3c6",
							"spark.synapse.context.notebookname": "roster_queue_processor",
							"spark.synapse.context.activityname": "roster_queue_processor",
							"spark.livy.synapse.skipSplitCodeExecution.enabled": "true",
							"livy.rsc.synapse.statement-output-for-error.enabled": "true",
							"spark.synapse.nbs.runid": "e829800c-7916-402a-97a6-034eaf61a3c6",
							"spark.synapse.nbs.kernelid": "",
							"spark.livy.synapse.ipythonInterpreter.enabled": "true",
							"livy.rsc.synapse.ipython-interpreter.options.ZMQInteractiveShell.display_page": "True"
						}
					},
					"idleAt": "2023-11-10T14:30:15.2261249Z",
					"notStartedAt": null,
					"startingAt": "2023-11-10T14:29:45.0231091Z",
					"deadAt": null,
					"shuttingDownAt": null,
					"killedAt": null,
					"recoveringAt": null,
					"busyAt": "2023-11-10T14:30:35.3824527Z",
					"errorAt": null,
					"currentState": "busy"
				},
				"schedulerInfo": {
					"submittedAt": "2023-11-10T14:29:39.4173284Z",
					"queuedAt": "2023-11-10T14:29:39.4886343Z",
					"scheduledAt": "2023-11-10T14:29:44.7621735Z",
					"endedAt": null,
					"cancellationRequestedAt": null,
					"currentState": "Scheduled"
				},
				"pluginInfo": {
					"currentState": "monitoring",
					"preparationStartedAt": "2023-11-10T14:29:44.7797365Z",
					"resourceAcquisitionStartedAt": "2023-11-10T14:29:44.7968628Z",
					"submissionStartedAt": "2023-11-10T14:29:44.9291271Z",
					"monitoringStartedAt": "2023-11-10T14:29:45.0448475Z",
					"cleanupStartedAt": null
				},
				"registeredSources": [
					"default"
				],
				"computeName": "idrsynspark",
				"artifactId": "Livy",
				"jobType": "SparkSession",
				"tags": {}
			},
			"result": {
				"runId": "e829800c-7916-402a-97a6-034eaf61a3c6",
				"runStatus": "Failed",
				"message": "Notebook execution is in Failed state, runId: e829800c-7916-402a-97a6-034eaf61a3c6",
				"error": {
					"ename": "LivyHttpRequestFailure",
					"evalue": "Something went wrong while processing your request. Please try again later. HTTP status code: 500. Trace ID: 54c0067d-1d45-481a-9d9a-339a03585c14.",
					"traceback": [
						"LivyHttpRequestFailure: Something went wrong while processing your request. Please try again later. HTTP status code: 500. Trace ID: 54c0067d-1d45-481a-9d9a-339a03585c14."
					]
				},
				"sparkPool": "idrsynspark",
				"notebookSnapshotUrl": "....",
				"notebookSnapshotStudioUrl": "....",
				"exitValue": null,
				"errors": {
					"ename": "LivyHttpRequestFailure",
					"evalue": "Something went wrong while processing your request. Please try again later. HTTP status code: 500. Trace ID: 54c0067d-1d45-481a-9d9a-339a03585c14.",
					"traceback": [
						"LivyHttpRequestFailure: Something went wrong while processing your request. Please try again later. HTTP status code: 500. Trace ID: 54c0067d-1d45-481a-9d9a-339a03585c14."
					],
					"_deprecated": "Use \"error\" instead."
				},
				"sessionId": "2805",
				"lastCheckedOn": "2023-11-10T14:44:39.62Z"
			}
		},
		"Error": null,
		"ExecutionDetails": {}
	},
	"effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime (Central US)",
	"executionDuration": 904,
	"durationInQueue": {
		"integrationRuntimeQueue": 0
	},
	"billingReference": {
		"activityType": "ExternalActivity",
		"billableDuration": [
			{
				"meterType": "AzureIR",
				"duration": 0.26666666666666669,
				"unit": "Hours"
			}
		]
	}
}

Answer 1

@Bhatt, Ramachandra - Thanks for the question and using MS Q&A platform.

It seems like you are facing an error while running a Synapse notebook written in Python language. The error message indicates that the LivyHttpRequestFailure 500 error is occurring and the session is stopping abruptly from execution. This error message suggests that something went wrong while processing your request, and it is recommended to try again later.

This error message is indicating that there is an issue with the Livy service, which is responsible for managing the Spark sessions in Synapse. There could be several reasons for this error, such as network connectivity issues, resource constraints, or configuration issues.

To troubleshoot this issue, you can try the following steps:

This error can occur for a variety of reasons, such as network connectivity issues, resource constraints. Here are a few things you can try to resolve the issue:

• Wait and try again later: Sometimes, the issue may be temporary and can be resolved by waiting and trying again later. If the issue persists, try the next steps.
• Check the network connectivity between your Synapse workspace and the Spark cluster. Ensure that the network is stable and there are no network-related issues.
• Check the resource utilization: Check the resource utilization of your Azure Synapse Analytics workspace to ensure that you have enough resources available to run your job. You can check the resource utilization by going to the "Monitoring" section of your Synapse Analytics workspace.
• Check the job logs: Check the job logs to see if there are any error messages or warnings that can help you diagnose the issue. You can view the job logs by going to the "Monitor" section of your Synapse Analytics workspace studio and selecting the job that failed.
• Try running the same Spark job on a different Spark cluster to see if the issue is specific to the current cluster.
• If none of the above steps resolve the issue, please do share the (WorkspaceName, SparkPoolName, Spark Runtime Version, LivyJobId, ResourceUri, Region, Issue Start Time YYYY-MM-DD HH:MM (UTC) which will help to diagnose the issue and provide guidance on how to resolve it.

For more details, refer to Monitor your Apache Spark applications in Synapse Studio.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.