Auto loader

Vineet S 1,350 Reputation points
2024-11-20T09:02:26.5633333+00:00

Hi,

How to add new column automatically when it is not present in delta table

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,272 questions
{count} votes

Accepted answer
  1. Chandra Boorla 4,845 Reputation points Microsoft Vendor
    2024-11-21T02:17:24.28+00:00

    Hi @Vineet S

    Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

    To automatically add a new column to a Delta table when it is not present, you can enable schema evolution in your write operations. When schema evolution is enabled, any columns that are present in the source data but missing from the target Delta table will be automatically added during the write transaction.

    User's image For example, when using Auto Loader or a batch write operation, you can set the mergeSchema option to true. This will ensure that new columns from the source data are appended to the end of the Delta table schema.

    The following example demonstrates using the mergeSchema option with a batch write operation:

    (spark.read
      .table(source_table)
      .write
      .option("mergeSchema", "true")
      .mode("append")
      .saveAsTable("table_name")
    )
    

    For additional information, please refer the below Microsoft documentations:

    Update Delta Lake table schema

    How does Auto Loader schema evolution work?

    I hope this information helps. Please do let us know if you have any further queries.


    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    0 comments No comments

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.