Azure VMSS Autoscale Triggering Multiple Events and Causing Multiple Function Executions

Alexandru Ene 0 Reputation points
2024-09-17T08:38:41.96+00:00

I am facing an issue with Azure VMSS autoscale notifications (through webbhook) triggering multiple executions of an Azure Function. This causes my function, which includes a pipeline trigger and a DNS update, to be executed multiple times in parallel, leading to unintended behavior, such as triggering my Azure DevOps pipeline and DNS updates three times instead of once.

VMSS Configuration: The VMSS has autoscale enabled, and I am monitoring autoscale notifications to trigger actions after VM scale events (both scale-in and scale-out).

Azure Function: Language: PowerShell 7.4 Trigger: HTTP Trigger Purpose: The function processes autoscale notifications, checks the provisioning state of VMSS instances, triggers an Azure DevOps pipeline, and updates DNS records. Expected Behavior: Trigger the function once for each scale event (either scale-in or scale-out) and perform the VMSS provisioning check, trigger the pipeline once, and update DNS once.

When I manually execute the function code (outside the autoscale event), everything works as expected. The function runs only once, triggers the pipeline once, and performs the DNS update correctly.

The issue only occurs when the function is triggered automatically via the autoscale notification from VMSS, where it runs three times instead of once.

The same event is logged three times in Azure Function logs. Here is an example of the log entries:

2024-09-17T06:50:26.311 [Information] INFORMATION: Triggering DNS update function... 2024-09-17T06:51:00.320 [Information] INFORMATION: DNS update function triggered. Response: 2024-09-17T06:51:00.323 [Information] Executed 'Functions.ansible_pipeline' (Succeeded, Id=78b2f41a-6b7c-412d-b147-2f3632d7835f, Duration=66337ms) 2024-09-17T06:51:00.730 [Information] INFORMATION: DNS update function triggered. Response: 2024-09-17T06:51:00.730 [Information] Executed 'Functions.ansible_pipeline' (Succeeded, Id=b2048c9c-1c28-4ea7-b6d0-9f8217fa6f0d, Duration=86803ms) 2024-09-17T06:51:01.164 [Information] INFORMATION: DNS update function triggered. Response: 2024-09-17T06:51:01.165 [Information] Executed 'Functions.ansible_pipeline' (Succeeded, Id=8e0db316-ea98-4376-8af3-10dd8e080665, Duration=107182ms) INFORMATION: {"version": "1.0","status": "Activated","operation": "Scale Out","context": {"timestamp": "2024-09-16T15:42:41.0770601Z","id": .... INFORMATION: {"version": "1.0","status": "Activated","operation": "Scale Out","context": {"timestamp": "2024-09-16T15:42:41.0770601Z","id": .... INFORMATION: {"version": "1.0","status": "Activated","operation": "Scale Out","context": {"timestamp": "2024-09-16T15:42:41.0770601Z","id": ....

User's image

In the Azure Function logs, it shows that the autoscale event (e.g., Scale Out) is received multiple times with the same timestamp and details, causing the function to execute multiple times.

This is the json payload sent out by VMSS.

{
    "version": "1.0",
    "status": "Activated",
    "operation": "Scale Out",
    "context": {
        "timestamp": "2024-09-16T15:16:40.5377227Z",
        "id": "/subscriptions/sid/resourceGroups/rg-name/providers/microsoft.insights/autoscalesettings/auto-scale-set-vmss-name",
        "name": "auto-scale-set-vmss-name",
        "details": "Autoscale successfully started scale operation for resource 'vmss-name' from capacity '1' to capacity '2'",
        "subscriptionId": "sid",
        "resourceGroupName": "rg-name",
        "resourceName": "vmss-name",
        "resourceType": "microsoft.compute/virtualmachinescalesets",
        "resourceId": "/subscriptions/sid/resourceGroups/rg-name/providers/Microsoft.Compute/virtualMachineScaleSets/vmss-name",
        "portalLink": "https://portal.azure.com/#resource/subscriptions/sid/resourceGroups/rg/providers/Microsoft.Compute/virtualMachineScaleSets/vmss-name",
        "resourceRegion": "eastus",
        "oldCapacity": "1",
        "newCapacity": "2"
    },
    "properties": {}
}

Probably the VMSS sent 3 notification like "scale up initiated", "scale up in progress" and "scale up completed".

Replication steps:

  1. VMSS auto scales (scales out) based on autoscale rules.
  2. An HTTP-triggered Azure Function is set to process the autoscale notification.
  3. The function logs multiple identical notifications for the same autoscale event, causing it to be executed three times for a single scale-out event.
  4. As a result, the Azure DevOps pipeline and DNS update operations are also triggered three times, which is undesirable.
Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
3,266 questions
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,990 questions
Azure Virtual Machine Scale Sets
Azure Virtual Machine Scale Sets
Azure compute resources that are used to create and manage groups of heterogeneous load-balanced virtual machines.
398 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. MayankBargali-MSFT 70,441 Reputation points
    2024-09-19T07:26:09.3466667+00:00

    @Alexandru Ene Thanks for reaching out.

    In you function app code you should only review for the status as completed and further execute your business flow. For all three events your configured webhook (i.e. Azure function app) will gets triggered.

    If you are only interested on a specific one when the operation is completed, then you should check the payload and validate if the status is completed (please double check the value) and execute your further workflow.

    Feel free to get back to me if you have any queries or concerns.

    Please 'Accept Answer' if it helped so that it can help others in the community looking for help on similar topics.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.