Function app randomly fails

Question

I have a durable function app with 4 functions, which suppose to process a task that takes about 10 to 15 minutes to complete. However, it fails randomly, when multiple requests are inbound to the application. Our scenario is like following:

• A request is inbound to an orchestrator that passes the parameters to the durable functions.
• The first function creates a directory in tmp file system that the Azure Function app provides, and passes the directory path to the second function.
• The second function executes some long logics on the files located in the directory path.

In the scenario, the second function fails to recognize the directory path that the first function passed to it. More specifically, the path value is passed, but the second function says it doesn't exist. What would be the possible cause, and how can I resolve this?

Answer 1

@Kay Park The issue you are encountering could be due to a number of reasons.

1. The temporary file system (/tmp) provided by Azure Functions is not persistent across different function executions. This means that even if you create a directory in the first function, it might not be available for the second function if they run in different instances.
2. Concurrency issues might be another factor. When multiple requests are being processed, they might be handled by different instances of the function app. Each instance has its own isolated /tmp directory, so the second function might not be able to access the directory created by the first function if they are running on different instances.
3. Durable Functions manage state by serializing and deserializing data between function invocations. If the directory path is being passed as part of this state, it's important to ensure that the path is valid and accessible when the second function runs.

In terms of solutions assuming you are impacted by one of the above root causes, I would suggest the below steps.

1. Instead of relying on the local /tmp file system, consider using durable entities or external storage like Azure Blob Storage to store and share data between function executions. This ensures that the data is accessible regardless of which instance the functions are running on.

      // Example of using Azure Blob Storage
      CloudStorageAccount storageAccount = CloudStorageAccount.Parse(connectionString);
      CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
      CloudBlobContainer container = blobClient.GetContainerReference("my-container");
      CloudBlockBlob blob = container.GetBlockBlobReference("my-directory-path.txt");
      // Function 1: Write directory path to blob
      await blob.UploadTextAsync(directoryPath);
      // Function 2: Read directory path from blob
      string directoryPath = await blob.DownloadTextAsync();
      
      
   

2. Pass the necessary data through the durable function's context instead of relying on the file system.

      // Function 1
      public static async Task<string> Function1([ActivityTrigger] string input, ILogger log)
      {
          string directoryPath = CreateDirectoryInTmp();
          return directoryPath;
      }
      // Function 2
      public static async Task Function2([ActivityTrigger] string directoryPath, ILogger log)
      {
          if (!Directory.Exists(directoryPath))
          {
              throw new Exception("Directory does not exist.");
          }
          // Process files in directory
      }
      // Orchestrator
      public static async Task RunOrchestrator([OrchestrationTrigger] IDurableOrchestrationContext context)
      {
          string directoryPath = await context.CallActivityAsync<string>("Function1", null);
          await context.CallActivityAsync("Function2", directoryPath);
      }
      
      
   

3. Ensure that the directory creation logic is idempotent, meaning it can be safely called multiple times without changing the result. This can help in scenarios where the directory might get cleaned up and needs to be recreated.

      // Function 1
      public static string CreateDirectoryInTmp()
      {
          string directoryPath = Path.Combine(Path.GetTempPath(), "my-directory");
          if (!Directory.Exists(directoryPath))
          {
              Directory.CreateDirectory(directoryPath);
          }
          return directoryPath;
      }
      
      
   

Keep in mind I have to make some assumptions based on the details you shared. If you require further assistance, please share with us as many details as you can so we can narrow down the cause.

Otherwise, by considering these solutions, you can make your durable function app more reliable and resilient to random failures when handling multiple requests.