How to Monitor the Data added to ADLS2 Storage accounts in daily basis?

Question

Hello, I am looking for an option to check the data ingested to ADLS2 storage accounts on a daily basis.
for overall Capacity, I can go the Storage Browser but I am looking for data added to storage account on particular date.

Answer 1

Hello @Goud, Madhu !

Welcome to Microsoft QnA!

you can utilize Azure Monitor :

Here's an example query to retrieve the data ingested to your ADLS Gen2 storage account on a daily basis:

StorageBlobLogs

| where TimeGenerated > ago(1d)

| where OperationName == "PutBlob"

https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-best-practices?wt.mc_id=knwlserapi_inproduct_azportal#monitor-telemetry

As Azure CoPilot is saying :

To enable Azure Storage logs in Azure Monitor through the Azure portal, you can follow these steps:

1. Navigate to the Azure portal and sign in with your Azure account.
2. In the left-hand menu, click on "Storage accounts".
3. Select the storage account for which you want to enable logs.
4. In the storage account menu, click on "Diagnostic settings".
5. Click on "+ Add diagnostic setting".
6. In the "Diagnostic setting" form, provide a name for the setting.
7. Check the "Archive to a storage account" box and select the storage account where you want to save the logs.
8. In the "Log" section, select the logs you want to enable.
9. Click on "Save".

Once you have enabled the logs, you can view them in Azure Monitor.

For more detailed instructions, you can refer to the following Azure documentation:

• Enable and manage Azure Storage Analytics logs (classic)
• Monitoring Azure Blob Storage
• Monitoring Azure Queue Storage
• Monitoring Azure Table storage

--

I hope this helps!

The answer or portions of it may have been assisted by AI Source: Azure Co Pilot

Kindly mark the answer as Accepted and Upvote in case it helped!

Regards

Answer 2

@Goud, Madhu Thanks for posting your query on Microsoft Q&A.

To monitor the data ingested to your Azure Data Lake Storage Gen2 account daily, you can use Azure Monitor Logs and create a query to retrieve the information you need. Here's a general outline of how you can do this:

Enable logging for your storage account:

Go to your Azure Storage account in the Azure portal.

Under Monitoring, select "Diagnostic settings" and then "Add diagnostic setting".

Enable logging for "Read" and "Write" operations.

Access logs in Azure Monitor:

Go to Azure Monitor in the Azure portal. Navigate to Logs and open a new query.

Write a query to filter data by date:

Use the AzureDiagnostics table to query the storage account logs. Use the StorageLogs table if you are using the older version of diagnostic settings. Filter by OperationName (e.g., "PutBlob", "AppendBlob") and the timestamp (TimeGenerated) to get data added on a particular date.

Aggregate the data:

You can use sum or count functions to aggregate the data size based on your need. Group by the ResourceGroup or StorageAccountId if you have multiple storage accounts.

Visualize the data:

You can visualize the ingested data using charts or tables in the Azure Monitor query editor. Here's a basic example query to get you started:

AzureDiagnostics
| where TimeGenerated >= datetime(2024-03-29) and TimeGenerated < datetime(2024-03-30)
| where OperationName == "PutBlob" or OperationName == "AppendBlob"
| summarize IngestedDataBytes = sum(ToDouble(RequestedContentLength_s)) by ResourceGroup

This query filters the logs for a specific date (2024-03-29 in this case), and aggregates the data ingested (RequestedContentLength_s) by the ResourceGroup. Adjust the date and aggregation as needed for your scenario.