Monitoring Action - Webhook Alert Response 403

Question

I've setup an Monitoring Webhook Alert that fires based on Metrics Usage (Ex: API Usage is > 1000). The Action group associated to this Alert Rule has been set with an endpoint

Now when I do a "Test Action Group" with any sample type, out of 9/10 times I get 403 Forbidden and worked only once. Now it just takes longer time spew out 403 response error.

I can confirm that the webhook URL is reachable and just works when I do the following from any shell

curl -X POST "https://www.my-homeAssistant-instance.com/api/webhook/movie-untitled-snagged"

Edit: Home Assistant accepts JSON data and the following is that 1/10 call's request

trigger:
  platform: webhook
  webhook_id: 'movie-untitled-snagged-....'
  json:
    schemaId: azureMonitorCommonAlertSchema
    data:
      essentials:
        monitoringService: CostAlerts
        firedDateTime: '2023-06-02T13:10:02.702Z'
        description: >-
          Your spend for budget Test_actual_cost_budget is now $11,111.00
          exceeding your specified threshold $25.00.
        essentialsVersion: '1.0'
        alertContextVersion: '1.0'
        alertId: >-
          /subscriptions/11111111-1111-1111-1111-111111111111/providers/Microsoft.CostManagement/alerts/Test_Alert
        alertRule: null
        severity: null
        signalType: null
        monitorCondition: null
        alertTargetIDs: null
        configurationItems:
          - budgets
        originAlertId: null
      alertContext:
        AlertCategory: budgets
        AlertData:
          Scope: /subscriptions/11111111-1111-1111-1111-111111111111/
          ThresholdType: Actual
          BudgetType: Cost
          BudgetThreshold: $50.00
          NotificationThresholdAmount: $25.00
          BudgetName: Test_actual_cost_budget
          BudgetId: >-
            /subscriptions/11111111-1111-1111-1111-111111111111/providers/Microsoft.Consumption/budgets/Test_actual_cost_budget
          BudgetStartDate: '2022-11-01'
          BudgetCreator: ******@sample.test
          Unit: USD
          SpentAmount: $11,111.00

Service: Azure Maps

Subscription: Gen2

Answer 1

Here are a few things you can check to troubleshoot this issue:

Verify the webhook URL: Ensure that the webhook URL you have configured in the Azure Monitor alert action group is correct and accessible. Double-check for any typos or missing characters.

Check authentication requirements: If your webhook endpoint requires authentication, make sure you have correctly configured the authentication details (e.g., API keys, headers, tokens) in the Azure Monitor action group. Confirm that the authentication method matches the requirements of your endpoint.

Test the webhook URL independently: Use tools like cURL or Postman to send a test request to your webhook URL outside of the Azure Monitor context. Verify that the URL is accessible and returns the expected response. This step will help isolate the issue to the Azure Monitor configuration.

Review firewall and network settings: Check if there are any firewall rules or network restrictions in place that might be blocking requests from the Azure Monitor service. Ensure that the necessary ports and protocols are open to allow incoming connections.

Check endpoint response requirements: Ensure that your webhook endpoint is correctly configured to handle incoming requests from Azure Monitor. Verify that it can accept the specific payload format and HTTP method (POST) used by Azure Monitor. You can refer to the Azure Monitor documentation for information on the expected payload structure.

Review access control and permissions: Ensure that the user or service principal associated with the Azure Monitor action group has the necessary permissions to access and call the webhook URL. Check the authentication and authorization settings for your webhook endpoint to ensure it allows requests from Azure Monitor.

Answer 2

Hi @SpiritBreaker ,

The only thing I can think of; based on your screenshot, is change your Action Type to Secure Webhook. Your endpoint is https but your using http for the action group.

I would also confirm the JSON payload being sent to the webhook to ensure the endpoint URI is able to process the payload. However, I would expect an 50x but still worth looking into.