IsDelta = True, Merge gets error "Not a delta table"

Ryan Abbey 1,181 Reputation points
2021-07-19T23:24:14.517+00:00

I'm running a Spark 2.4 cluster for which we've created a delta table and the function IsDelta returns true for the table (well, path), however, when trying to run an merge statement, we are getting the error "<table> is not a Delta table.". The table is referenced via its path for the merge so the creation/validation and merge are all via the path method but it does give error as "table" rather than "path", not sure if that makes a difference

create statement
df.write.format("delta").mode("overwrite").option("path", path).saveAsTable(table)

merge statement
deltaUpdate.merge(...)
deltaUpdate.whenNotMatchedInsertAll()
deltaUpdate.execute()

Merge statement works fine as has been used on multiple other tables without issue.

If I query the table via either spark.sql or %%sql, they both query the table fine giving indications that the table is created properly so why am I getting the error and how to get around it (short of re-creating the table)?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,919 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Ryan Abbey 1,181 Reputation points
    2021-07-20T00:23:09.817+00:00

    The error giving the table name was relevant... the calling procedure was passing the table rather than the path


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.