write the data into the in-memory table created in the databricks

Question

I am trying to write the data into the in-memory table created in the databricks DB using the below command. but the issue is.. it is not appending the data instead everytime inserting new records only and deleting the old records.

orderInputDF = (orderInputDF
.writeStream
.format("memory")
.queryName(orders_table)
.outputMode("append")
.start()
)

Answer 1

Hello @Akshay tondak ,

Thanks for the question and using MS Q&A platform.

This is an expected behaviour when you use Append mode: this is the default mode. Just the new rows are written to the sink.

The output can be configured in different modes:

• Complete Mode: The entire updated result table is written to external storage. It is up to the storage connector to decide how to handle the writing of the entire table.
• Append Mode: Only new rows appended in the result table since the last trigger are written to external storage. This is applicable only for the queries where existing rows in the Result Table are not expected to change.
• Update Mode: Only the rows that were updated in the result table since the last trigger are written to external storage. This is different from Complete Mode in that Update Mode outputs only the rows that have changed since the last trigger. If the query doesn’t contain aggregations, it is equivalent to Append mode.

For more details, refer to Structured Streaming tutorial and Azure Databricks - Table streaming reads and writes.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators