Load files in appending in databricks

Shambhu Rai 1,406

Hi Expert,

I have 2 files in blob storage .

file1.csv, file2.csv

how will i load it in a view using appending order using *.csv extension using temp view

file_list = dbutils.fs.ls("/mnt/mount/test/file")

1 answer

Suba Balaji 11,186

Hi,

You can try the below code - (I am assuming, the files are under test/file folder)

final_files_list=[]
file_list = dbutils.fs.ls("/mnt/mount/test/file")
for file in file_list:
  final_files_list.append(file.name)
df = spark.read.load(path=final_files_list,format="csv", sep=",", inferSchema="true", header="true")
df.createOrReplaceTempView("TestFiles")

We are creating a pyspark data frame from the given csv files, and convert that to a temp view called TestFiles.

Please try and let us know for questions.

Thanks

Shambhu Rai 1,406 Reputation points

2023-11-29T10:14:01.0166667+00:00

getting error: path does not found or exist
Shambhu Rai 1,406 Reputation points

2023-11-29T10:24:01.1566667+00:00

i want to load file data not path
Suba Balaji 11,186 Reputation points

2023-11-29T10:48:46.1166667+00:00

Can you post your full code? Have you mounted your blob storage on data lake already?

Shambhu Rai 1,406

i need date filter on the blob storage file before add into it.. your query is working directly on blob storage path

from  pyspark.sql.functions import current_timestamp 
from pyspark.sql.functions import *
from pyspark.sql.types import *
from datetime import *
import pandas as pd
from pyspark.sql.types import IntegerType,StringType,LongType,StructField,StructType
schema= StructType([
        StructField("path", StringType()),
        StructField("name", StringType()),
        StructField("size", IntegerType()),
        StructField("moditime", LongType()),
    ])
=============================
file_list = dbutils.fs.ls("dbfs:/mnt/mount/test/test")
display(file_list)
============================
file_list = dbutils.fs.ls("dbfs:/mnt/mount/test/test")
display(file_list)
==============================

df=df.select('path','name','size',(df.moditime/1000).cast('timestamp').alias('Updatedtime'))


========================
df=df.select('path','name','size','Updatedtime').filter(df['Updatedtime']>=
current_timestamp()).show()

Shambhu Rai 1,406 Reputation points

2023-11-30T12:32:49.06+00:00

suggestion pls