Parquet file to table in databricks

Question

Hi Expert, How we can connect to new paraquet file using table which already load using same kind of file before Mint/mnt2/paraquet1 this was first Delta file. Now I got new file Mint/mnt2/paraquet2 for same table how we can connect it

Answer 1

@Shambhu Rai - Thanks for the question and using MS Q&A platform.

To connect to a new Parquet file using a table that was already loaded using the same kind of file before, you can follow these steps:

Load the new Parquet file into a DataFrame:

df = spark.read.parquet("Mint/mnt2/parquet2")

Create a temporary view of the DataFrame:

df.createOrReplaceTempView("new_table")

Use the INSERT INTO statement to insert the data from the new table into the existing table:

spark.sql("INSERT INTO existing_table SELECT * FROM new_table")

This will insert the data from the new table into the existing table. You can then drop the temporary view:

spark.catalog.dropTempView("new_table")

Please note that the above steps assume that you have already created the existing_table and loaded data into it from the first Parquet file. If you have not done so, you will need to create the table and load data into it before following the above steps.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.