Deploying mutiple Functions Apps

Question

Deploying mutiple Functions Apps

Jona 925

Hi everyone,

I've deployed many functions apps in my consumption plan. The functions are Python's ones, version 3.10, running on Linux Platform on Azure.

These functions are listening to an Event Hub and pushing messages to ADLS. This is the code

const { app, output } = require('@azure/functions');
const { functionHandler } = require('./includes')

const availableTables = [
    "direccion",
    "detalle-tabla-dividendo",
    "estado",
    "mutuo-cia",
    "tabla-dividendo"
];

for (const table of availableTables) {

    // Output to Event Hubs
    const eventHubOutput = output.eventHub({
        eventHubName: table,
        connection: `EventHubConnection.${table}`,
    });

    // Listening to Blob Storage
    app.storageBlob(table, {
        path: process.env[`StoragePath.${table}`],
        connection: `StorageConnection.${table}`,
        return : eventHubOutput,
        source: 'EventGrid',
        handler: functionHandler
    });
}

I set my settings for these functions to wait 1 minute in case a batch is not full.

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request"
      }
    }
  },
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[4.*, 5.0.0)"
  },
  "extensions": {
    "eventHubs": {
      "maxEventBatchSize" : 50,
      "minEventBatchSize" : 40,
      "maxWaitTime" : "00:01:00",            
      "batchCheckpointFrequency" : 1,
      "initialOffsetOptions" : {
          "type" : "fromStart"
      },
      "clientRetryOptions":{
          "mode" : "exponential",
          "tryTimeout" : "00:01:00",
          "delay" : "00:00:00.80"
      }
    }
  }
}

And here is my issue/question. When I enable App Insights, I realized that just one function is waiting for the time to get a batch. Essentially, all the functions should be waiting, but this is not happening. This is App Insights:

Last-Modified:Wed, 13 Dec 2023 06:31:59 GMT

2023-12-13T06:59:00Z [Information] Request [bd9d954f-9ef4-4146-9abe-4e0a4849b017] PUT https://XXXXXXX.blob.core.windows.net/azure-webjobs-hosts/locks/landing-rdz/host?comp=lease x-ms-lease-action:renew x-ms-lease-id:629f8a27-e025-76cb-41c3-7c5a73932803 x-ms-version:2021-08-06 Accept:application/xml x-ms-client-request-id:bd9d954f-9ef4-4146-9abe-4e0a4849b017 x-ms-return-client-request-id:true User-Agent:azsdk-net-Storage.Blobs/12.13.0 (.NET 6.0.24; Microsoft Windows 10.0.20348) x-ms-date:Wed, 13 Dec 2023 06:58:59 GMT Authorization:REDACTED client assembly: Azure.Storage.Blobs

------- Just one function is refreshing Auth to Event Hub
2023-12-13T06:59:04Z [Information] Beginning refresh of AMQP link authorization for Event Hub: detalle-tabla-dividendo (Service Endpoint: 'amqps://XXXXXXX.servicebus.windows.net/detalle-tabla-dividendo').

2023-12-13T06:59:04Z [Information] Completed refresh of AMQP link authorization for Event Hub: detalle-tabla-dividendo (Service Endpoint: 'amqps://XXXXXXX.servicebus.windows.net/detalle-tabla-dividendo').
--------

2023-12-13T06:59:11Z [Information] Request [5541637c-c8e2-492d-81e3-70046c8c4b46] HEAD https://XXXXXXX.blob.core.windows.net/azure-webjobs-hosts/locks/landing-rdz/host x-ms-version:2021-08-06 Accept:application/xml x-ms-client-request-id:5541637c-c8e2-492d-81e3-70046c8c4b46 x-ms-return-client-request-id:true User-Agent:azsdk-net-Storage.Blobs/12.13.0 (.NET 6.0.24; Microsoft Windows 10.0.20348) x-ms-date:Wed, 13 Dec 2023 06:59:11 GMT Authorization:REDACTED client assembly: Azure.Storage.Blobs


------- Just one function is waiting time to get message from Event Hub
2023-12-13T06:59:20Z [Verbose] Poll for function 'detalle-tabla-dividendo' on queue 'azure-webjobs-blobtrigger-landing-rdz' with ClientRequestId 'b07194a5-cb42-4eac-9e41-5ef9156bb42c' found 0 messages in 4 ms.


2023-12-13T06:59:20Z [Verbose] Function 'detalle-tabla-dividendo' will wait 25482.1918 ms before polling queue 'azure-webjobs-blobtrigger-landing-rdz'.
----------

2023-12-13T06:59:21Z [Information] Request [c58a8b00-21a0-4501-a03c-72db1c3948d9] HEAD https://XXXXXXX.blob.core.windows.net/azure-webjobs-hosts/locks/landing-rdz/host x-ms-version:2021-08-06 Accept:application/xml x-ms-client-request-id:c58a8b00-21a0-4501-a03c-72db1c3948d9 x-ms-return-client-request-id:true User-Agent:azsdk-net-Storage.Blobs/12.13.0 (.NET 6.0.24; Microsoft Windows 10.0.20348) x-ms-date:Wed, 13 Dec 2023 06:59:21 GMT Authorization:REDACTED client assembly: Azure.Storage.Blobs


2023-12-13T06:59:41Z [Information] Response [a39bf6cb-d417-4eb3-80d5-45d1f0734577] 200 OK (00.0s) Accept-Ranges:bytes ETag:"0x8DBFBA5360B1FD0" Server:Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0 x-ms-request-id:380d40d9-a01e-0058-5391-2db2a1000000 x-ms-client-request-id:a39bf6cb-d417-4eb3-80d5-45d1f0734577 x-ms-version:2021-08-06 x-ms-resource-type:REDACTED x-ms-meta-FunctionInstance:REDACTED x-ms-creation-time:Wed, 13 Dec 2023 05:47:05 GMT x-ms-lease-status:locked x-ms-lease-state:leased x-ms-lease-duration:fixed x-ms-blob-type:BlockBlob x-ms-server-encrypted:true x-ms-access-tier:Hot x-ms-access-tier-inferred:true x-ms-owner:REDACTED x-ms-group:REDACTED x-ms-permissions:REDACTED Date:Wed, 13 Dec 2023 06:59:40 GMT Content-Length:0 Content-Type:application/octet-stream Content-MD5:1B2M2Y8AsgTpgAmY7PhCfg== Last-Modified:Wed, 13 Dec 2023 06:31:59 GMT

2023-12-13T06:59:41Z [Information] Request [ad29a76a-234a-4711-a736-f211a14afa35] PUT https://XXXXXXX.blob.core.windows.net/azure-webjobs-hosts/locks/landing-rdz/host?comp=metadata x-ms-meta-FunctionInstance:REDACTED x-ms-lease-id:d2534bf5-a920-b54d-3805-05141df1da6a x-ms-version:2021-08-06 Accept:application/xml x-ms-client-request-id:ad29a76a-234a-4711-a736-f211a14afa35 x-ms-return-client-request-id:true User-Agent:azsdk-net-Storage.Blobs/12.13.0 (.NET 6.0.24; Microsoft Windows 10.0.20348) x-ms-date:Wed, 13 Dec 2023 06:59:41 GMT Authorization:REDACTED client assembly: Azure.Storage.Blobs

2023-12-13T06:59:41Z [Information] Response [ad29a76a-234a-4711-a736-f211a14afa35] 200 OK (00.0s) ETag:"0x8DBFBA9146EA026" Server:Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0 x-ms-request-id:380d40e6-a01e-0058-5f91-2db2a1000000 x-ms-client-request-id:ad29a76a-234a-4711-a736-f211a14afa35 x-ms-version:2021-08-06 x-ms-request-server-encrypted:true Date:Wed, 13 Dec 2023 06:59:40 GMT Content-Length:0 Last-Modified:Wed, 13 Dec 2023 06:59:41 GMT

2023-12-13T06:59:41Z [Information] Host lock lease acquired by instance ID 'd2534bf5a920b54d380505141df1da6a'.


------- Just one function is waiting time to get message from Event Hub
2023-12-13T06:59:46Z [Verbose] Poll for function 'detalle-tabla-dividendo' on queue 'azure-webjobs-blobtrigger-landing-rdz' with ClientRequestId '9fdc35a4-3d6e-467e-af82-b60531d7b788' found 0 messages in 5 ms.

2023-12-13T06:59:46Z [Verbose] Function 'detalle-tabla-dividendo' will wait 53551.0197 ms before polling queue 'azure-webjobs-blobtrigger-landing-rdz'.
----------

It's like just one functions would be respecting the settings. However, I tested the others functions and works, but don't wait time to get a full batch not refreshing Auth to Event hub.

Since they are not waiting for a full batch, the only way to trigger them is push more files onto ADLS to fill the batch.

Could somebody help me understand what is happening here?

Regards

MayankBargali-MSFT 70,996 Reputation points Moderator

2023-12-26T17:04:24.96+00:00

@Jona Thanks for reaching out and apology for the inconvenience due to this issue. Can you please confirm if you are still facing the issue.
Jona 925 Reputation points

2023-12-26T18:02:16.56+00:00

Still facing it

1 answer

Your answer

MayankBargali-MSFT 70,996 Reputation points Moderator

2023-12-26T17:04:24.96+00:00

@Jona Thanks for reaching out and apology for the inconvenience due to this issue. Can you please confirm if you are still facing the issue.
Jona 925 Reputation points

2023-12-26T18:02:16.56+00:00

Still facing it

Answer 1

MayankBargali-MSFT 70,996 Moderator

@Jona As per the document here maxWaitTime is define as The maximum interval that the trigger should wait to fill a batch before invoking the function. This would be applicable for the trigger and not to output binding. I see that you have storage blob input binding and event hub output binding.

Jona 925 Reputation points

2023-12-27T14:31:47.8066667+00:00

Sorry, It was my typo ... I refer to the second group of Functions (Event Hub -> ADLS).

Thanks for pointing me out
MayankBargali-MSFT 70,996 Reputation points Moderator

2023-12-28T05:40:36.6266667+00:00

@Jona Thanks for sharing more details. In case of event hub trigger it is not guaranteed as documented on the same link.

The minimum size isn't strictly guaranteed. A partial batch is dispatched when a full batch can't be prepared before the maxWaitTime has elapsed. Partial batches are also likely for the first invocation of the function after scaling takes place.
Jona 925 Reputation points

2023-12-28T15:41:03.5333333+00:00

It's something that was already aware and knew it. My question is other.

If you look at the logs, you'll see that just one functions is respecting maxWaitTime and going to event hub looking for more messages.

This behaviour should be present in all functions, no just one.

regards
MayankBargali-MSFT 70,996 Reputation points Moderator

2024-01-02T06:46:48.34+00:00

@Jona I will suggest you to reach out to review it over one on one support request to validate it. In case if you don't have the support plan then please let me know.
Jona 925 Reputation points

2024-01-03T02:52:45.2766667+00:00

I'll appreciate it. Let me perform a final test to be sure of this strange behaviour.

If it persists, I would be gratefull to have a support ticket.
MayankBargali-MSFT 70,996 Reputation points Moderator

2024-01-03T11:51:39.2466667+00:00

@Jona Sure. Let me know what was the outcome.

Share via

Deploying mutiple Functions Apps

1 answer

Your answer