Host Restarts in Azure Function Triggered by EventHub, Causing Data Loss

Jusuf Ramic 0 Reputation points
2024-09-11T09:51:55.3466667+00:00

Hello,

I'm currently working with a ~v4 Azure Function running on Linux, implemented in python 3.9 (using v2 programming model), which is triggered by EventHub. The function processes incoming payloads, performs transformations, and writes the data into an SQL Server database by executing stored procedures, using pyodbc library.

Issue: Occasionally, the function fails in such a way that it restarts the host, bypassing the retry functionality. Looking at Application Insights, the failed invocation includes the message "Invocation cancelled - exiting retry loop". Sometimes this happens even after the function executes all steps successfully and prints the final log. This causes data loss when the failure occurs before the SQL Server procedure execution, since once the host restarts, the same event is not reprocessed. Of course, if the failure happens after the procedure execution, no data loss is observed.

Details:

  • Runtime: Python 3.9
  • Azure Function: written using v2 programming model
  • App Service Plan: B2: 1
  • Trigger: EventHub
  • Target: SQL Server (using pyodbc to execute stored procedures)
  • The function has retry logic in place, but host restarts circumvent this mechanism, leading to lost events.
  • The failures occur during increased load, and is most likely related to database transactions, even though sometimes it occurs after the transaction is successfully completed
  • Looking at "Diagnose and solve problems" section seems to point to error 139 (segmentation fault) - python exited with error 139 (0x8B)
  • The failures are always accompanied by the following trace in app insights: Exceeded language worker retry restart count for runtime: python. Shutting down and proactively recycling the Functions Host to recover.
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,999 questions
Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
641 questions
SQL Server
SQL Server
A family of Microsoft relational database management and analysis systems for e-commerce, line-of-business, and data warehousing solutions.
13,820 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Ryan Hill 28,186 Reputation points Microsoft Employee
    2024-09-12T19:47:52.1833333+00:00

    Hi @Jusuf Ramic.

    The python 139 error, there's a troubleshooting guide that you can take a look at. See Troubleshoot: python exited with code 139 for steps you can take to mitigate the issue. If none of them don't help, please do let me know.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.