Sink failure -Output to single file - No such file

Question

Hi All,

I have several pipelines setup to extract data from SQL tables/views and export the data into CSV files which are being pushed out to an SFTP location.

Within the sink settings I have the following configured:

Output to a single file for which the file name is being dynamically assigned as follows:

concat('accountView_', toString(currentDate()), '.csv')

Under the Optimize tab:

Everything has been deployed for at least 6-8 months and running as expected producing files daily.

However, as of yesterday all pipelines have started to fail with the following error:

When re-running the failed task I notice that some folders are created on the SFTP:

There are few other folders contained within but I was not able to capture all of them. I think they named something like .temporary/0

When the pipeline fails these folders are no longer present on the SFTP.

I had double checked that the SFTP had enough space the account can still create files.

I wasn't able to immediately find anything too helpful online so made a copy of the original pipeline and dataflow to check over the settings.

From the testing I have done so far (which is mainly with adjusting the settings with the sink). I've noticed that the pipeline will now only succeed with the following settings:

None of the other options under "File name option" or "Partition option" seem to be working.

If any other setting is selected, the pipeline will fail with the same error message as shared above.

I cannot figure out why this is all of a sudden.

If anyone has any thoughts? Suggestions it would be appreciated.

In the mean time I will continue my investigations.

Thank you for reading.