This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 53%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EML%3C/text%3E%3C/svg%3E)
I am having this error frequently in daily processing when processing PySpark notebook. Can anyone tell me what it could be?
This notebook is being called through a pipeline.
Hi @Marines Lopes ,
Welcome to Microsoft Q&A platform and thanks for posting your question here.
Could you please share the pipeline execution log from Monitoring tab > Apache Spark Applications
It would also be helpful to see the notebook code so I can try to reproduce the error. Looking forward to your response. Thankyou
Hi @Marines Lopes ,
Just checking if you got a chance to look at the previous response. Requesting you to share more relevant information so that we can help you better. Thanks
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 75%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERP%3C/text%3E%3C/svg%3E)
Hello @AnnuKumari-MSFT , I'm work with @Marines Lopes ,
We noticed that the notebook actually runs successfully (in the apache spark applications log).
Error is always in the pipeline and intermittent, sometimes it happens, sometimes it doesn't. And it's a simple pipeline.
Here are log of spark notebook execution
and here are the log 252663-spark31s10v-22166-application-1666278791213-0024-d.log
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 62%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESB%3C/text%3E%3C/svg%3E)
We are getting the same issue !