dataflow - updateif --> true()

arkiboys 9,621

In dataflow I have source1, alter row, sink1
alter row setting is using : update if --> true()
This process is working fine.
Sink1 is .parquet file and on each load I can see that the lastmodified date column of this .parquet file has been updated.
I would like to find-out, whether the file is re-created or just updated.
How can I find out what is happening to the file? is it updated or is it re-created?

Thank you

MartinJaffer-MSFT 26,021 Reputation points

2021-09-30T21:13:51.653+00:00

Hello @arkiboys . Could you tell me what the implications are of being re-created versus updated?

Also, if every row in the file is changed, is it still the same file? Would it still be updating, rather than creating a new one?
While asking whether something is updated, or re-created sounds simple, there is actually more grey area when we look at the mechanism of writing data.

Therefore, to get a more useful answer, I am wondering what difference is made between the two for you.

Accepted answer

arkiboys 9,621 Reputation points

2021-10-01T06:40:56.603+00:00

That is my question. How do I find out if the system is updating or re-creating the file?
Please sign in to rate this answer.
MarkKromer-MSFT 5,186 Reputation points Microsoft Employee

2021-10-03T06:52:04.66+00:00

If your sink is a blob store or ADLS and you're writing Parquet files, then Alter Row is ignored. Alter Row only works for database sinks. To update Parquet files, use Delta Lake format instead.

arkiboys 9,621 Reputation points

2021-10-03T07:49:43.19+00:00

I see.
I will try this out and come back to you.
Thank you
Sign in to comment

dataflow - updateif --> true()

0 additional answers