Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,309 questions
Synapse Spark Notebook runs manually fine - times out in Pipeline. Why?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 39%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Mark Mace
71
Reputation points
The Spark Notebook in question:
If ran manually, takes 2.5 minutes and finishes successfully
If in a pipeline, errors out after 2 hours, with below error message
"errorCode": "6002", "message": "Error: Job aborted.\norg.apache.spark.sql.errors.QueryExecutionErrors$.jobAbortedError
Looking at the pipeline running, it's spinning in circles on the last step, which is an INSERT.
- I've tried:
- Running the Notebook manually (as said works fine; finishes in 2.5 minutes)
- Running the Notebook manually, as managed identity (works the same - fine)
- Creating a brand new pipeline with only the troublesome Spark Notebook in it - that pipeline also fails after 2 hours (timeout I'm guessing)?
- Running pipeline on a large Spark pool - at 24 minutes and counting, so pretty sure this is going to fail too.
Thoughts?
Similar Post: https://www.reddit.com/r/AZURE/comments/15sv3es/synapse_spark_noteobook_runs_fine_normally_times/
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 38%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQ%3C/text%3E%3C/svg%3E)
QuantumCache 20,356 Reputation points2023-08-16T20:58:03.4066667+00:00 Hello @Mark Mace
I believe Spark pool is configured correctly and has the necessary resources to run the job? right?
And the timeout settings are not too low and that they allow enough time for the job to complete?
What activity is done in your Notebook? anything can we try to repro from our side?
How about checking creating a simple Notebook to get Current UTC time and test the scenario?
Or Remove the Notebook and run the Pipeline?
So that we can take out the doubt on Notebook from the equation?
-
Mark Mace 71 Reputation points
2023-08-16T21:15:00.1433333+00:00 Hi,
Maybe my description was bad?
The Spark Notebook - runs great, if I run it manually. It takes about 2 and 1/2 minutes.
If I put that Spark Notebook, into a pipeline, the pipeline fails (seemingly timing out after 2 hours). It was failing in the pipeline it originally was in, and is failing in the same way, if I create a brand new pipeline just for the single Spark Notebook.
The bit that it's hanging up on in the notebook, is an INSERT.
Does that answer your questions?
-
QuantumCache 20,356 Reputation points
2023-08-16T21:27:28.1666667+00:00 Hello @Mark Mace,
"timing out after 2 hours," is definitely not an ideal scenario!
We may need more info from your side to investigate or repro your scenario!
I am trying to understand what exactly the Notebook is executing (INSERT ?)? any SQL script? or inline code? can you please share more details? if possible share screenshots?
What is the Spark Pool configuration ?
-
Mark Mace 71 Reputation points
2023-08-16T21:33:55.96+00:00 Hi,
Attached is the last insert statement (spark sql), and the spark pool.
It is a small spark pool, with default settings. I also tried with a medium.
The thing is, it runs fine (with a small spark pool) in the Spark Notebook itself. It's only when put in a Pipeline that it fails.
-
QuantumCache 20,356 Reputation points
2023-08-25T18:03:21.43+00:00 Hello @Mark Mace Just checking if we are still connected on this discussion? Please let us know if you need to add more info so that we better assist you!
-
Mark Mace 71 Reputation points
2023-08-25T18:32:13.4333333+00:00 I had replied with the requested info, and was waiting for a response.
Though - in the iterim, the spark notebook when run manually, is now also failing. So maybe the root issue isn't the pipeline / can close this item.
Sign in to comment
Sign in to answer