ADLS Gen2 - Slow File List

Kyle M 1 Reputation point
2022-06-16T15:00:29.263+00:00

We're using an ADLS gen2 blob container for our ETL pipeline.

There's one folder in particular that is our "landing" point where we have new files being generated (5 / hour) via small append operations. Those files get moved off and processed, but we usually only have ~5-10 files in that folder at any given time.

Over time, this folder has become increasingly slow to open (list the files) via storage explorer and the azure portal console.

Viewing the folder took ~70 seconds just now. Sometimes it's slower, sometimes its faster. Is this a known issue? No other folders have this issue.

I'm wondering if we should be avoiding this usage pattern in ADLSv2 and create new folders instead of re-using the same one? Did I miss something in the documentation?

Thanks.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,355 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,721 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,449 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Kyle M 1 Reputation point
    2022-06-16T15:24:28.187+00:00

    Note, I just recreated the folder and it loads instantly again. Not a great long term solution, but at least I have a workaround.