Getting Keyboard interrupt error while executing pipeline in Azure Synapse Analytics

Question

Hello,

I am using Azure Synapse Analytics to execute a pipeline that loads data from source tables into target tables within an Azure SQL Database. The pipeline follows a parent-child notebook architecture, where the main parent notebook is responsible for triggering child notebooks listed in the metadata tables. However, while the pipeline is running, I encounter a KeyboardInterrupt error during the execution of one of the child notebooks. I am currently unable to identify the root cause of this issue and would appreciate any help or guidance regarding the same.

Also, I am adding the error trace for the same below:

KeyboardInterrupt while sending command. Traceback (most recent call last):   File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/java_gateway.py", line 1038, in send_command     response = connection.send_command(command)   File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/java_gateway.py", line 1217, in send_command     answer = smart_decode(self.stream.readline()[:-1])   File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/socket.py", line 705, in readinto     return self._sock.recv_into(b) KeyboardInterrupt--------------------------------------------------------------------------- KeyboardInterrupt                         Traceback (most recent call last) Cell In[17], line 18      15 TargetTable = (pd_Metadata.loc[i, "TargetTable"])                              17 #Phase 3 ---> 18 Phase3Status = mssparkutils.notebook.run(Notebook,<>,{"ETLId": ETLId,"ObjectId": ObjectId,"ParentPipelineRunId":ParentPipelineRunId,"ChildPipelineRunId":ChildPipelineRunId,"LastLoadDate":LastLoadDate,"LandingTable":LandingTable,"SourceId":SourceId,"Notebook":Notebook})      20 ObjectStartTimeStamp = datetime.utcnow()      22 #Phase 4 File ~/cluster-env/clonedenv/lib/python3.10/site-packages/notebookutils/common/logger.py:32, in deprecated.<locals>.wrapper(*args, **kwargs)      30 def wrapper(*args, **kwargs):      31     if not deprecated_message_enabled or product_type != constants.TRIDENT: ---> 32         return func(*args, **kwargs)      33     module = str(func.module).split(".")[-1]      34     new_module = f"notebookutils.{module}" if module != "mssparkutils" else "notebookutils" File ~/cluster-env/clonedenv/lib/python3.10/site-packages/notebookutils/mssparkutils/notebook.py:21, in run(path, timeout_seconds, arguments, workspace)      19 @deprecated      20 def run(path, timeout_seconds=90, arguments={}, workspace=""): ---> 21     return notebook.run(path, timeout_seconds, arguments, workspace) File ~/cluster-env/clonedenv/lib/python3.10/site-packages/notebookutils/notebook.py:13, in run(path, timeout_seconds, arguments, workspace)      12 def run(path, timeout_seconds=90, arguments={}, workspace=""): ---> 13     exit_val = nb.run(path, timeout_seconds, arguments, workspace)      15     if exit_val == constants.STOP_SESSION_REQUEST_EXIT_VAL:      16         nb.exit(exit_val) File ~/cluster-env/clonedenv/lib/python3.10/site-packages/notebookutils/mssparkutils/handlers/notebookHandler.py:88, in SynapseNotebookHandler.run(self, path, timeout_seconds, arguments, workspace)      86     display_dag_task = threading.Thread(target=self.display_dag_async, args=(dagId, {}))      87     display_dag_task.start() ---> 88 j_notebook_run_result = self.notebookutils.run(      89     path, timeout_seconds, arguments, workspace, context)      90 exit_val = j_notebook_run_result.exitVal()      91 if display_progress_enabled:      92     # wait for the display_dag_async to finish File ~/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/java_gateway.py:1321, in JavaMember.call(self, *args)    1314 args_command, temp_args = self._build_args(*args)    1316 command = proto.CALL_COMMAND_NAME +\    1317     self.command_header +\    1318     args_command +\    1319     proto.END_COMMAND_PART -> 1321 answer = self.gateway_client.send_command(command)    1322 return_value = get_return_value(    1323     answer, self.gateway_client, self.target_id, self.name)    1325 for temp_arg in temp_args: File ~/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/java_gateway.py:1038, in GatewayClient.send_command(self, command, retry, binary)    1036 connection = self._get_connection()    1037 try: -> 1038     response = connection.send_command(command)    1039     if binary:    1040         return response, self._create_connection_guard(connection) File ~/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/java_gateway.py:1217, in GatewayConnection.send_command(self, command)    1213     raise Py4JNetworkError(    1214         "Error while sending", e, proto.ERROR_ON_SEND)    1216 try: -> 1217     answer = smart_decode(self.stream.readline()[:-1])    1218     logger.debug("Answer received: {0}".format(answer))    1219     if answer.startswith(proto.RETURN_MESSAGE): File ~/cluster-env/clonedenv/lib/python3.10/socket.py:705, in SocketIO.readinto(self, b)     703 while True:     704     try: --> 705         return self._sock.recv_into(b)     706     except timeout:     707         self._timeout_occurred = True KeyboardInterrupt:

Answer 1

Here are some effective steps to troubleshoot the KeyboardInterrupt error in Azure Synapse notebook execution:

Sometimes the issue is transient. Rerunning the pipeline may resolve temporary problems, such as service hiccups or brief compute unavailability.

Ensure that your Spark pool or Azure SQL Database is not under pressure. Resource exhaustion (e.g., memory, CPU) can lead to stalled execution or timeout-related interrupts.

The most probable cause of the KeyboardInterrupt is that mssparkutils.notebook.run() has a default timeout of 90 seconds. If a child notebook takes longer, the parent forcibly interrupts it, resulting in a KeyboardInterrupt.

Set an appropriate timeout:

mssparkutils.notebook.run("ChildNotebookPath", timeout_seconds=1800, arguments={...})

Review the exact section of the notebook where the error occurs, focusing on operations that may cause delays or instability. This includes long-running SQL queries, data skews or expensive shuffle operations within Spark, and external network or service calls that could hang or timeout. Identifying and optimizing these bottlenecks can help prevent execution interruptions like KeyboardInterrupt.

Ensure that the notebook is executed under a user or service context that has the necessary permissions to access all required resources. This includes access to the appropriate Spark pools, linked services such as Azure SQL or Blob Storage, and the necessary workspace-level permissions to run notebooks and interact with associated data sources.

If you continue encountering this issue, it might be helpful to gather more details about your configuration and the steps leading up to the error. Here are some follow-up questions you might consider:

• What are the configurations of the Azure Synapse pipeline and the child notebooks (e.g., timeout settings, resource allocation)?
• Are there any particular operations or SQL commands in the child notebooks that seem to take longer than expected?
• Have you noticed if this error occurs consistently with a specific notebook, or is it intermittent across different executions?
• Could you provide information on the Azure Synapse resource limits you are operating within?
• Are there any logs or error messages that indicate resource constraints?