Share via

LivyHttpRequestFailure: Something went wrong while processing your request. Please try again later. HTTP status code: 500

Pednekar, Pratik 0 Reputation points
2023-12-04T18:55:51.8466667+00:00

Hi Team,

While running a pyspark notebook in synapse pipeline. We are getting below error:

User's image

We are getting this error inconsistently in our one of the environment when we are trying to run job with large number of files.

We also tried to see error in spark job application, but we are not able to trace any error inside monitor.

Please help us in identifying the issue due to which we are getting mentioned error for LivyHttpRequestFailure.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. AnnuKumari-MSFT 34,561 Reputation points Microsoft Employee Moderator
    2023-12-05T07:09:49.0533333+00:00

    Hi Pednekar, Pratik ,

    Welcome to Microsoft Q&A platform and thanks for posting your query here.

    I understand that you are getting "LivyHttpRequestFailure" while running pyspark notebook in Synapse workspace.

    It seems that this error occurred because there is huge input data to process and assigned apache spark pool might be insufficient to process such amount of data.

    Could you please let us know what is the max node for your apache spark?

    Kindly try to increase the number of nodes on apache spark pool and Enable AutoScaling .

    Try choosing XL node size , if the issue still persists, try XXL

    For more details, kindly check out the following documentation:

    Node size of Apache Spark pool

    AutoScale for Apache spark pool

    Automatically scale Azure Synapse Analytics Apache Spark pools

    Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.