last modified date in databricks

Question

Hi,

Expert,

How to Pickup the files using last modified date from blob storage using databricks.. if receives 2 file in 2 mins interval how it will load it

/mnt/delta//Test.csv"

Answer 1

First you need to mount your Azure Blob storage to Databricks to access files in the blob storage. You can do this using the dbutils.fs.mount() method and precise the storage account name, container name, and access key.

Once it is done, you can use butils.fs.ls() to list all files in a directory, it will return a list of FileInfo objects, each containing details like path, name, and last modified time.

To filter them based on the last modified time, you can write a function that compares the last modified time of each file with the desired timestamp.

If you receive files every two minutes , think about setting up a scheduled job in Databricks that runs every 2 minutes.

# Mount blob storage
dbutils.fs.mount(source="blob storage path", mount_point="/mnt/my_mount_point", extra_configs={"fs.azure.account.key.<storage-account-name>.blob.core.windows.net":"<access-key>"})
# Function to filter and load files
def load_recent_files():
    files = dbutils.fs.ls("/mnt/my_mount_point/")
    for file in files:
        if file.lastModified >= desired_timestamp:  # Replace 'desired_timestamp' with your condition
            df = spark.read.csv(file.path)
            # Process and load the dataframe as needed
# NB: Don't forget to schedule this function to run at your desired interval

Answer 2

Hi Shambhu Rai,

You can consider creating a function and having parameters for file name and file and call that function every time to create a view.

Hope this helps. Please let me know if any further queries.

---

Please consider hitting Accept Answer button. Accepted answers help community as well.