Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,227 questions
Received Error: Spark_Ambiguous_NonJvmUserApp_FailedContainerLaunch in Synapse
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 71%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJZ%3C/text%3E%3C/svg%3E)
Jia Zhang
60
Reputation points
Hi Team,
In our Synapse pipeline, we have a spark notebook keep running for getting streaming data from eventhub. Yesterday, it was running for 4 hours, and suddenly failed due to this message:
Operation on target our-pipeline-name failed: Spark_Ambiguous_NonJvmUserApp_FailedContainerLaunch: Livy session has failed. Session state: Dead. Error code: Spark_Ambiguous_NonJvmUserApp_FailedContainerLaunch. [plugins.biot-prod-mtssext-cc-platform-synapse-ws.spark1.2302 WorkspaceType:<Synapse...... Livy Id=[181] Job failed during run time with state=[dead]. Source: Unknown.
The last log are:
stderr:
and there is a warnning:
2024-02-27 01:35:12,422 WARN AzureEventHubEmitter [SparkObservabilityBus-0]: message size larger than limit (dropped: 149 limit: 1048576 bytes)
stdout:
May I know what could caused this Spark_Ambiguous_NonJvmUserApp_FailedContainerLaunch issue? Thanks so much!
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bhargava-MSFT 31,216 Reputation points Microsoft EmployeeFeb 28, 2024, 10:14 PM Hello Jia Zhang, This is due to an outage in East US 2 affecting all jobs in the region.
PG has acknowledged the issue, and they are actively working on it. I will provide an update as soon as I hear something from them.
Thank you for your patience
-
Jia Zhang 60 Reputation points
Feb 28, 2024, 10:39 PM Thanks so much for! But just to confirm I'm in Toronto, will this also get affected? The issue is happened around 7pm-8:35pm, Feb 26th EST time. During this period our other job was running without error, only this one.
-
Jia Zhang 60 Reputation points
Feb 28, 2024, 10:40 PM Thanks so much for your answer! But just to confirm I'm in Toronot, will this outage also influence it? The issue is happened around 7pm-8:35pm Feb 26th EST time, during this time our other pipeline is working normal. Thanks!
-
Bhargava-MSFT 31,216 Reputation points Microsoft Employee
Feb 29, 2024, 12:08 AM If your ADF/synapse is deployed in the East US2 region, you will see this error. PG is currently working on this issue. I will provide an update as soon as I hear something from them.
-
Jia Zhang 60 Reputation points
Feb 29, 2024, 3:31 AM Thanks so much! Our synapse worksapce is in Canada Central, just wondering if it is also related to the outage? Thanks!
-
Bhargava-MSFT 31,216 Reputation points Microsoft Employee
Feb 29, 2024, 7:06 PM Hello Jia Zhang, Spark issue was mitigated last night. You shouldn't see this error anymore. In case you are still experiencing the issue, please let me know.
-
Bhargava-MSFT 31,216 Reputation points Microsoft Employee
Mar 4, 2024, 8:37 PM Hello Jia Zhang,
I am checking to see if you have any further questions here.
-
Jia Zhang 60 Reputation points
Mar 8, 2024, 9:09 PM Thanks so much!
But in recent few days, the pipeline is still having issue. This Synapse Spark pipeline is reading stream data from Eventhub, process data and send to another eventhub. Recent few days, it still running normal for few days, then suddenly failed, and all these three failure could be fix by just rerun the pipeline.
The error message for second time error at Feb 29:
Operation on target mobile-gen-ingestion-pipeline-spark failed: LivyHttpRequestFailure: Failed to send request to Livy. Please try again. If this issue still exists, please contact support. --> TaskCanceledException: The operation was canceled. ---> IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.. ----> SocketException: The I/O operation has been aborted because of either a thread exit or an application request.
The error message for third time error at Mar 4th:
Operation on target mobile-gen-ingestion-pipeline-spark failed: InvalidHttpRequestToLivy: Submission failed due to error content =[] HTTP status code: 408. Trace ID: 2a802b2a-ee3b-4adf-9fa2-3a13530f5179.
May I know if all three error are related to the infrastructure issue or is our pipeline facing some kind of issue? Thanks so much!
*For the service location, our service is in Canada Central instead of East US 2, may I know if Canada Central also infect by the issue?
If you need more detail for the last 2 errors I have uploaded the log here, please check
-
Jia Zhang 60 Reputation points
Mar 12, 2024, 3:04 PM Hi Team,
May I konw if we can get an update for this issue? Thanks so much!!
-
Bhargava-MSFT 31,216 Reputation points Microsoft Employee
Mar 12, 2024, 5:59 PM Hello Jia Zhang,
The issue seems to be transient in nature. I would suggest opening a support case for further investigation. In case you don't have a support plan, please let me know so that I can provide a one-time free support request to address this.
Sign in to comment
Sign in to answer