How to stream usage log from azure devops organization to storage account/log analytics workspace?

Question

is it possible to stream azure devops usage log to log analysis workspace or storage account similarly to audit logs?

Answer 1

Hi Piotr Piwowar

For Azure DevOps usage logs (distinct from audit logs), there isn't a single dedicated API endpoint labeled exactly "usage logs."

While there isn't a dedicated API for exporting usage logs, you can leverage the existing Azure DevOps REST API to gather relevant data.

You can use the Usage Monitoring API to track pipeline usage, top commands, and user activity.

Example API endpoint:

GET https://dev.azure.com/{organization}/_apis/usage?api-version=6.0

This API provides details on top users, commands, and pipelines.

Depending on the type of usage data you want to extract, there are various REST APIs that can give you relevant information about activities, pipelines, work items, etc.

Usage logs you can export via API:

• Use the Builds - List API to get pipeline/build run info.
• The Work Items - Query API lets you query work item changes
• Use the Test Runs API to get data on test executions.

Additional References:

https://stackoverflow.com/questions/64403158/download-audit-logs-in-csv-from-azure-devops-ui-using-powershell

Hope this helps!

Please Let me know if you have any queries.

If you found the information helpful, please click "Upvote" on the post to let us know and consider accepting the answer as the token of appreciation. Thank You.

Answer 2

Hi Piotr Piwowar

Yes, you can stream Azure DevOps usage logs to log analytics workspace or a storage account, similar to audit logs.

However, usage logs such as pipeline usage, job duration, or agent utilization metrics are not natively streamed to a Log Analytics Workspace or Storage Account in the same way as audit logs.

Log analytics workspace:

Go to Azure DevOps -> Organization Settings -> Policies -> Enable the option of Audit Logs.

Then Refresh the browser -> Now you should be able to see Auditing in Organization Settings.

Go to streams -> Click on New stream -> Select the stream target that you want to configure.

For Azure Monitor logs, firstly create Log Analytics Workspace in Azure Portal. Refer this document for creation - https://learn.microsoft.com/en-us/azure/azure-monitor/logs/quick-create-workspace?tabs=azure-portal

After creation, go to Agents in workspace and copy the workspace id and primary key.

Now Go to Azure Monitor Logs Stream -> Paste the workspace id and primary key -> Click on set up.

Storage account:

Event Grid supports Storage Queue as targets.

Firstly, create Storage Account in Azure Portal, Refer this document - https://learn.microsoft.com/en-us/azure/storage/common/storage-account-create?tabs=azure-portal

After creation of Storage Account -> Go to Queues -> Create a new Queue.

Now, create Event Grid in Azure Portal. Go to Event Grid -> Custom Events -> Topics -> Create. Refer this document for creation - https://learn.microsoft.com/en-us/azure/event-grid/create-custom-topic

Once Event Grid is created, go to Overview -> copy the topic endpoint

Go to Settings -> Access Keys -> copy the key.

Now Go to Streams in Azure DevOps -> Select Azure Event Grid -> paste the topic endpoint and access key.

To allow the Event Grid Resource to communicate with the Azure Storage Account Queue, I’m using Azure Managed Identity.

Navigate back to the Event Grid Topic created earlier and select Identity under the settings. Next, select System Assigned, which creates a 1-to-1 mapping between the Event Grid Resource and its Azure Identity Service Principal object. Save the changes.

Navigate back to the Azure Storage Account you created earlier -> Access Control (IAM) -> Role: Storage Queue Data Contributor -> Assign access to: User, Group, Service Principal and select Event Grid.

Now Go to Event Grid -> Entities -> Event Subscription -> Create by providing the endpoint details i.e. Storage Account, queue and managed identity as system assigned.

Then you can be able to see the events in Event Grid. And also, you can be able to see messages in storage account queues.

Additional References:

https://learn.microsoft.com/en-us/azure/devops/organizations/audit/auditing-streaming?view=azure-devops

Hope this helps!

Please Let me know if you have any queries.

If you found the information helpful, please click "Upvote" on the post to let us know and consider accepting the answer as the token of appreciation. Thank You.

Answer 3

Is there API where i could export usage log (not audit log) ?