sink Delta parquet error

Question

sink Delta parquet error

arkiboys 9,706

Any suggestions how to solve this issue?
Error
{"message":"Job failed due to reason: at Sink 'sinkDeltaUpdate': org.apache.spark.sql.AnalysisException: cannot resolve source.Nameid in search condition given columns target....`
Thank you

Accepted answer

0 additional answers

Your answer

Answer 1

ShaikMaheer-MSFT 38,551 Microsoft Employee Moderator

Hi @arkiboys ,

Thank you for posting query in Microsoft Q&A Platform.

From error message, It seems Nameid Column from source is not available. Could you please check it once?

Hope this will help. If not, More details and implementation and exact step where error is coming will help better to understand the case. If possible, kindly share more details on implementation and error where exactly its occurring?

---------------------------

Please consider hitting on Accept Answer if this helps.

arkiboys 9,706 Reputation points

2021-11-27T08:11:44.213+00:00

When having the merge option checked in the sink Delta parquet, should it not solve this issue so that when a new columnName is coming in then it simply gets added?
ShaikMaheer-MSFT 38,551 Reputation points Microsoft Employee Moderator

2021-11-29T14:38:42.207+00:00

Hi @arkiboys - When merge schema option is enabled, any columns that are present in the previous stream but not in the Delta table are automatically added on to the end of the schema.

Could you please make sure "Allow schema drift option" enabled and "validate schema" option disabled.

Kindly share more details of implementation, so that we can try same to repro scenario and help with resolution. Thank you,

Share via

sink Delta parquet error

0 additional answers

Your answer