invalidhttprequesttolivy: request body too large (limit payload to Livy 25 Kb)

alexander tikhomirov 31

Hello
I am trying to use input param for spark notebook activity in synapse. My param is string and about 18Kb.
But I have failed notebook execution due to this error. In this case what exactly payload is? It is documented somewhere?

Saurabh Sharma 23,821 Reputation points Microsoft Employee

2022-02-26T00:27:07.527+00:00
Hi @alexander tikhomirov ,

Thanks for using Microsoft Q&A!!
If I understand your scenario correctly, you are getting error "invalidhttprequesttolivy: request body too large. Limit for payload size is 25 Kb" while trying to run parameterized Spark notebook from Synapse using Spark notebook activity.
I have tested similar scenario while passing a string parameter of around 100KB, but it is working as expected and notebook is running fine without any errors. (Please see the screenshot below).

I am using the dbutils.widgets.get to read the parameter in the Spark notebook. Please find below the full code which I have used to test the same -

string_parameter = dbutils.widgets.get("parameter") print ("Param -\'string_parameter':") print (string_parameter)

Could you please help providing us more details on what exactly you are trying to do in Databricks notebook using the value received through the parameter?

Also, I am using a Spark cluster with the configuration - Driver: Standard_DS3_v2, Workers: Standard_DS3_v2, 0 workers, 10.3 (includes Apache Spark 3.2.1, Scala 2.12) for the Notebook.
Could you please share your configuration so that I can check with the same configuration in my environment ?

Please let me know if you have any other questions.

Thanks
Saurabh
alexander tikhomirov 31 Reputation points

2022-02-26T18:40:26.703+00:00

are you sure that you use synapse/notebook actvity? I support you are using databricks/notebook activity

I am using standard feature like base parameters to parametrize my spark notebook activity in synapse + special cell to get this param

with my approach and with your suggestion the same error, and could you provide where in official documentation of synapse provided information about dbutils.widgets.get() for spark notebook, as I understand it is about databricks.

my spark pool config
Saurabh Sharma 23,821 Reputation points Microsoft Employee

2022-03-01T00:31:33.757+00:00

Hi @alexander tikhomirov ,
Yes, you are right. I have checked against Databricks notebook.
However, based on your information, I have tried checking against a Synapse notebook activity, and it also worked as expected. (Please see below).

I have used the same spark cluster configuration as yours.

Did you try to dig further into your apache pool logs like I have checked in the above screenshot.

Also, could you please tell me in which region you Synapse workspace is located?

Thanks
Saurabh
alexander tikhomirov 31 Reputation points

2022-03-01T18:08:31.343+00:00

west europe

my steps in and logs in attachements
178991-doc-error.pdf

178870-sparktesttitan1-618-application-1646149034359-0005.log
Saurabh Sharma 23,821 Reputation points Microsoft Employee

2022-03-01T18:16:11.457+00:00

Hi @alexander tikhomirov ,
Thanks for the details. I will check against a west Europe instance and get back to you.

Thanks
Saurabh
Saurabh Sharma 23,821 Reputation points Microsoft Employee

2022-03-11T15:59:26.747+00:00

Hi @alexander tikhomirov ,
Sorry for the delay on this one. I am having some problems with the creating the instance in West Europe due to my subscription limitations. I suggest you to please open an Azure Support ticket so that someone can look into your issue in detail. In case you have any limitations opening a support ticket please let me know and I will help you providing one time free support ticket.

Thanks
Saurabh
alexander tikhomirov 31 Reputation points

2022-04-29T11:12:05.237+00:00

@Saurabh Sharma maybe there are some updates or ideas?

update from my side, still have some error when trying to use as input some data for spark activity, for both, notebook and job
see picture

Share via

invalidhttprequesttolivy: request body too large (limit payload to Livy 25 Kb)

Your answer