How to Report File Metadata for 2000+ Files Uploaded by Multiple Users to Azure Blob Storage

Question

How to Report File Metadata for 2000+ Files Uploaded by Multiple Users to Azure Blob Storage

Prasad Sandu 0

Hi Team,

I would like to know following questions. Please do needful.

1.How to Report File Metadata for 2000+ Files Uploaded by Multiple Users to Azure Blob Storage

2.How to connect Hadoop hive to databricks

Regards,

Prasad,

5017654887.

1 answer

Your answer

Answer 1

Amira Bedhiafi 34,101 Volunteer Moderator

Hello Prasad !

Thank you for posting on Microsoft Learn.

You can use ADF with ADLS or blob storage. For ADF, you can use the Get Metadata activity to retrieve metadata for each file and you can then use Foreach activity with pagination or batching.

Then you build reports in Power BI or Azure Synapse Analytics.

For the destination, you can use Azure SQL DB, CSV, or Parquet in Data Lake.

If you are comfortable with Azure functions or python scripting in Azure SDK

from azure.storage.blob import BlobServiceClient
service = BlobServiceClient.from_connection_string("<your-connection-string>")
container_client = service.get_container_client("<your-container-name>")
for blob in container_client.list_blobs():
    print(blob.name, blob.size, blob.last_modified, blob.metadata)

Then you export results to CSV or push to a SQL database and you can automate this via Azure Functions or ADF custom activity.

When it comes to connecting Hive to Databricks and querying Hive tables:

Export Delta Lake data to a Hive-compatible format (Parquet or ORC).
Mount ADLS Gen2 or Blob Storage in Hive using ABFS or WASB protocol.
Create external Hive tables on top of those files.

Or from Databricks to Hive :

Option 1: Use Unity Catalog (recommended for governance).
Option 2: Use an external Hive metastore (from HDInsight, Cloudera).

spark.conf.set("hive.metastore.uris", "thrift://<hive-host>:9083")
spark.sql("SHOW TABLES IN default").show()

Prasad Sandu 0 Reputation points

2025-07-03T12:03:54.7933333+00:00

Multiple users are uploading multiple different files in blob storage, i need to extract metadata like who uploaded by,location,time,file type,file size. how to to get this.

Filename | Size | Uploaded By | Upload Time | Location|File type

Also i know ,hadoop hive connection to Azure databricks.

Usually each workspace has own access connector, how to setup this
Venkat Reddy Navari 3,470 Reputation points Microsoft External Staff Moderator

2025-07-04T13:30:11.3066667+00:00
Hi Prasad Sandu Thanks for the follow-up and the extra context helpful.

How to Report Metadata for Files in Azure Blob Storage (Including “Uploaded By”)

You're right that getting basic metadata like filename, size, and last modified is easy. But tracking who uploaded the file isn’t available by default it needs to be captured via logging or tagging during upload.

Option A: Use Azure Storage Logs to Track Upload Info

You can enable Storage Account diagnostic settings to send blob operation logs (like BlobCreated) to Log Analytics or a Storage blob container.

These logs include fields like:

identity.claims.upn → shows who uploaded the file (if authenticated via Azure AD)

Upload timestamp, file path, etc.

Once logs are in place, you can:

Use ADF, Databricks, or Logic Apps to parse them

Extract: FileName | Size | UploadedBy | UploadTime | FileType | Path

Store it in a table (Azure SQL, CSV, or Power BI)

Official guide: https://learn.microsoft.com/en-us/azure/storage/blobs/monitor-blob-storage?tabs=azure-portal

Option B: Add Uploader Info as Metadata During Upload

If users are uploading through: A custom app, or Azure Function, or Logic App,

You can attach user info as blob metadata at the time of upload:

blob_client.set_blob_metadata({'uploaded_by': '******@contoso.com'})

This gives you full control and is ideal for future tracking.

How to Connect Hadoop Hive to Azure Databricks

Just to summarize the two common approaches:

If you're using an external Hive metastore (e.g. from HDInsight or Cloudera):

spark.conf.set("hive.metastore.uris", "thrift://<your-hive-host>:9083") spark.sql("SHOW TABLES").show()

Or use Unity Catalog (if available in your workspace):

Helps unify governance and works with Delta Lake

No need to manage separate Hive metastores

For either method, make sure:

You’ve got network access to the Hive Thrift server (if external)

Permissions are correctly set up (especially with Unity Catalog)

Reference: https://docs.azure.cn/en-us/databricks/archive/external-metastores/external-hive-metastore?utm_source

Hope this helps. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

How to Report File Metadata for 2000+ Files Uploaded by Multiple Users to Azure Blob Storage

1 answer

Your answer