java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]

Question

I've been trying to execute a python script for a crawler using the Apache Spark pool in Synapse. When I start the spark job execution, 3-4 mins into the job , after a few function executions, I get the following error log and the script terminates:

21/11/23 14:31:50 ERROR ApplicationMaster: Uncaught exception:
java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:220)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:469)
at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:305)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply$mcV$sp(ApplicationMaster.scala:245)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:780)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:779)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:244)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:804)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)

Any suggestions on how I can get this fixed? The same code seems to be running fine on my local python env.

Note: I am not using parallel execution or any of the spark features. It is a plain python script which makes request calls.

Answer 1

Hello @Swapnil kumar and welcome to Microsoft Q&A. I will try to explain the error message as best I can.

SIGNAL TERM is a signal used to tell the operating system to end the process. There are multiple possible causes for this , usually an unhandled exception.
The next line gives further insight:

Final app status: FAILED, exitCode: 13, (reason: User application exited with status 143)  

If I focus on the 13, this could be a broken pipe signal, and the default action on broken pipe is to terminate the process.

If I focus on the 143, this is a status code with which your code chose to stop with. When applications end, they return a status code. If I recall correctly, 0 is the default and means everything ran as expected. Any other value indicates an error happened. The number is dictated by your code. In this case you would search your code for the value 143, or search each of the exit commands.