Share via

Writing to the sink failing with error: Invalid column name 'true'

Subedi, Manish 40 Reputation points
2024-06-26T13:31:00.6866667+00:00

Hi,
One of our dataflow is failing in production with following error:
'Job failed due to reason: at Sink 'sink1': Invalid column name 'true'.

Invalid column name 'true'.

The dataflow inserts records to a Synapse SQL server table. This was working fine when last ran on May 31st but suddenly it starts failing.

There is nothing updated on the dataflow or to the table used in sink.

We also tried to write to a parquet file with same flow, same error.
Please note, when ran in debug mode-> data preview, it displays the records.

The column type used are: string, integer, boolean, timestamp

User's image

Looking forward for your advice.

SQL Server
SQL Server
A family of Microsoft relational database management and analysis systems for e-commerce, line-of-business, and data warehousing solutions.
13,164 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,977 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. BhargavaGunnam-MSFT 28,446 Reputation points Microsoft Employee
    2024-06-26T16:03:20.4733333+00:00

    Hello Subedi, Manish,

    Welcome to the Microsoft Q&A forum.

    After further checking with my internal team, the issue turned out to be due to a recent Spark upgrade. Could you please raise a support case so that a support engineer can downgrade the Spark version for your workspace?

    In case if you don't have a support plan, please let me know so that I can enable a one time free support request to work on this case.

    0 comments
  2. Subedi, Manish 40 Reputation points
    2024-06-27T19:56:44.31+00:00

    Thanks for your response.

    We recreated the new dataflow with same logic and it worked.
    Now it can write to the sink/table.

    Do you have the root cause for this issue, why just one job failed while others were running fine?

    Also, as there are no other jobs failure related to this spark upgrade, I think we are good for now.