ADLS Gen2 Access Logs via Diagnostic Settings

Question

Hi,

I was wondering if there is any way to enable access logs like read, write, delete on Azure Data Lake Storage Gen2 (ADLS Gen2). For Azure Blob Storage we achieved this via Diagnostic Settings, but to me it looks like this doesn't cover any operations made via the *.dfs.core.windows.net endpoint.

The Diagnostic Settings (classic) in the Portal are mentioning this:

Azure Data Lake Storage Gen2 logs are available only with version 2.0. Blob logs are supported in both versions 1.0 and 2.0. Learn more.

However, the classic Diagnostic Settings have no ability to effectively query the gathered logs. Does anyone has an idea how to enable the ADLS access logs via the modern Diagnostic Settings to bring them into Log Analytics ?

Answer 1

@Cedric Ahlers

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this!

Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer. Accepted answers show up at the top, resulting in improved discoverability for others.

Issue: How to enable the ADLS access logs via the modern Diagnostic Settings to bring them into Log Analytics?

Solution: Customer shared - "With the help of Microsoft Support we found the root cause why StorageDelete Events aren't logged to Log Analytics. In my case the Log Analytics Workspace was using a Data Collection Rule Transformation with a KQL Query that filtered specific API Operations.

The KQL Transformation looked like this:

source
| where OperationName in ("PutBlob", "PutBlock")
| extend [...]

so, the Diagnostic Settings were correctly sending the events to log analytics, but due to the filter in the transformation query only StorageWrite operations ended up in the StorageBlobLogs table."

---

If your issue remains unresolved or have further questions, please let us know in the comments how we can assist. We are here to help you and strive to make your experience better and greatly value your feedback.

Answer 2

Hi Cedric Ahlers - Thanks for reaching out over Q&A Forum.

ADLS Gen2 is a capability on top of BLOB Storage account so if you have configured the logging for blob storage, that should suffice for ADLS gen2 as well. The same is for classic logging too.

Now, the Blob API's can be used for ADLS Gen2 as well and in that case the endpoint shall reflect as that of Blob only.

If you are calling any ADLS Gen2 specific API such as CreateFile, AppendFile, FlushFile, ListPath etc. then the call shall happen over the DFS endpoint.

https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/operation-groups?view=rest-storageservices-datalakestoragegen2-2019-12-12

Since you have already enabled the logging, I would suggest reviewing the API's in there. If those are Blob specific API, that will explain.

You can tend to make some calls for ADLS Gen2 API via CURL and then observe the behavior ahead.

https://techcommunity.microsoft.com/t5/azure-paas-blog/performing-simple-adls-gen2-storage-rest-api-operations-using/ba-p/3491555

Hope that helps.

Answer 3

In case someone is facing the same problem.

With the help of Microsoft Support we found the root cause why StorageDelete Events aren't logged to Log Analytics. In my case the used Log Analytics Workspace was using a Data Collection Rule Transformation with an KQL Query that filtered specific API Operations.

The KQL Transformation looked like this:

source
| where OperationName in ("PutBlob", "PutBlock")
| extend [...]

so the Diagnostic Settings were correctly sending the events to log analytics, but due to the filter in the transformation query only StorageWrite operations ended up in the StorageBlobLogs table.