Azure Application Gateway Log Analyzer using GoAccess
IntroductionThis template configures the GoAccess log analyzer for Azure Application Gateway access logs. Using GoAccess, users can quickly analyze and view their Application Gateway statistics in real time using their browser through generated HTML reports.
The template creates an Ubuntu VM under your (customer) subscription, installs Apache HTTP web server as well as the GoAccess log analyzer, and then connects the VM with the customer’s Blob container to periodically fetch incremental access logs of Application Gateway. GoAccess will parse the access logs and display rich statistics on traffic.
By default, GoAccess installed by this template will parse and display traffic stats for the past 3 days’ worth of logs, if present.
- Access to an Azure subscription to deploy a Virtual machine with a Public DNS name.
- Enable access logging and store logs in desired storage account as specified here. Please also note the following:
- Only the ApplicationGatewayAccessLog will be used by GoAccess
- You want to make sure you are sending and storing your ApplicationGatewayAccessLog to a storage account (select the “Archive to a storage account” check box if using the Portal to enable Application Gateway logging).
- In your storage account container, ensure you have Shared Access Signature key configured. The expiry date time needs to be set to a date much further out in the future (eg: 1 year out from now). Also, only the Read and List permissions are needed for GoAccess. Make sure to generate the connection string as well. The Blob Container Service SAS URL connection string is what you need to input to the ARM template.
You can generate Service-level SAS URL for the Blob Container "insights-logs-applicationgatewayaccesslog" using Azure Storage Explorer for your operating system. Storage Explorer is available for Windows, MacOS and Linux.
For example, the blob container SAS URL should look like this -
Alternatively, you can generate the Service-level SAS using REST API. Read more about it here.
Running the ARM template
Click on “Deploy to Azure” button in the template Readme.md in the ARM template.
The template will require a set of parameters input from you as the user:
- adminUsername: Username you want to use for the VM the template creates
- adminPassword: Password you want to use to log in to the VM
- dnsNameForPublicIP: The DNS name (prefix) you want to use for the VM to map against its public IP
- appGwAccessLogsBlobSasUri: The SAS URL connection string (see 2(iii) in the Pre-requisites list above) for the storage account blog where your Application Gateway Access Logs are stored
- FilterRegexForAppGwAccessLogs: A regex to use to filter the Application Gateway Access Logs to a specific subset. For example, if you have multiple application gateways publishing logs to the same storage account blob, and you only want GoAccess to surface traffic stats for say one of the Application Gateways, you can provide a regex for this field to filter to just that instance.
- Region: The Azure region where you would like the VM to be created
Once the template deployment is successful, you can view the real time analytics by accessing the link http://Public-DnsNameOfVM/report.html where Public-DnsNameOfVM is the DNS name entered as input to the template.
User can view the logs based on the parameters available in the Application Gateway’s access logs. The GoAccess statistics that can be observed for Application Gateway are General Statistics, Unique Visitors, Requested files, Requested statics files, 404 or Not Found, Hosts, Operating Systems, Browsers, Visit Times, Virtual Hosts, Geo Location and HTTP Status Codes. For more details on these statistics please see the GoAccess man page.
Please note following aspects related to this template:
- There may be up to 5 minutes delay (beyond the latency in pushing logs from Application Gateways to Storage account) in seeing statistics updated on GoAccess.
- This solution may result in increased data, network, or compute resource usage in Azure. The solution may increase a customer’s Azure license or subscription costs.
- The time duration of logs that can be analyzed depends on the size of the RAM and disc capacity configured for the underlying VM.
- This VM periodically (every 24 hour) reports the health of the VM to the Microsoft. The heartbeat contains the compute metadata of the VM published by Azure's instance metadata service.
By default, the GoAccess dashboard and associated data are unsecured. Since the web server is Apache HTTP Webserver, you can secure access by following the Apache Auth documentation.
Also, since it is a Virtual Machine, you can use Network Security Groups to allow/deny IP addresses to restrict access, but make sure that outbound internet connectivity is allowed to reach the storage account.
For any issues with running this template, please file an issue in GitHub under Azure/azure-quickstart-templates repository: https://github.com/Azure/azure-quickstart-templates/issues
At this time no SLA is offered for support – this is strictly for use as-is, but we will do our best in responding to issues raised.
For any feature requests or general help with GoAccess itself, please file an issue in GitHub under the GoAccess repository: https://github.com/allinurl/goaccess/issues
GoAccess is distributed under the MIT License. For details of the licensing terms, please refer to the GoAccess License terms.
Apache HTTP Web Server is distributed under the Apache 2.0 License. For details of the licensing terms, please refer to the Apache 2.0 License terms.
Tags: Microsoft.Network/publicIPAddresses, Microsoft.Network/networkSecurityGroups, Microsoft.Network/virtualNetworks, Microsoft.Network/networkInterfaces, Microsoft.Compute/virtualMachines, Microsoft.Compute/virtualMachines/extensions, CustomScript