Share via

ADLS Gen2 Access Logs via Diagnostic Settings

Cedric Ahlers 15 Reputation points
2024-07-17T15:08:17.9566667+00:00

Hi,

I was wondering if there is any way to enable access logs like read, write, delete on Azure Data Lake Storage Gen2 (ADLS Gen2). For Azure Blob Storage we achieved this via Diagnostic Settings, but to me it looks like this doesn't cover any operations made via the *.dfs.core.windows.net endpoint.

The Diagnostic Settings (classic) in the Portal are mentioning this:

Azure Data Lake Storage Gen2 logs are available only with version 2.0. Blob logs are supported in both versions 1.0 and 2.0. Learn more.

However, the classic Diagnostic Settings have no ability to effectively query the gathered logs. Does anyone has an idea how to enable the ADLS access logs via the modern Diagnostic Settings to bring them into Log Analytics ?

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,423 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,905 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Amrinder Singh 4,270 Reputation points Microsoft Employee
    2024-07-17T16:05:11.19+00:00

    Hi Cedric Ahlers - Thanks for reaching out over Q&A Forum.

    ADLS Gen2 is a capability on top of BLOB Storage account so if you have configured the logging for blob storage, that should suffice for ADLS gen2 as well. The same is for classic logging too.

    Now, the Blob API's can be used for ADLS Gen2 as well and in that case the endpoint shall reflect as that of Blob only.

    If you are calling any ADLS Gen2 specific API such as CreateFile, AppendFile, FlushFile, ListPath etc. then the call shall happen over the DFS endpoint.

    https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/operation-groups?view=rest-storageservices-datalakestoragegen2-2019-12-12

    Since you have already enabled the logging, I would suggest reviewing the API's in there. If those are Blob specific API, that will explain.

    You can tend to make some calls for ADLS Gen2 API via CURL and then observe the behavior ahead.

    https://techcommunity.microsoft.com/t5/azure-paas-blog/performing-simple-adls-gen2-storage-rest-api-operations-using/ba-p/3491555

    Hope that helps.