Unable to Read Input Schema in Sink settings tab in Data Flow in Azure Data Factory

Question

I have a source, derived column , Sink transformation in my data flow.

Derived Column transformation , I am getting some Year and Month from Source.

But I cannot see these(Year and month) column names in Sink settings tab when using file name option as pattern. But these column names are available in Optimize Tab of Sink.

I want file names as fact_combined_pnl_$Year_$Month in the $Year/$Month folder. Please check following screenshots

Attaching Derived Column Settings, Sink Settings and Expression Builder, Sink Optimize

Please suggest

Answer 1

Hi @Krishnamohan Nadimpalli ,

Thank you for posting query in Microsoft Q&A Platform.

As per my understanding, you want to write expression for filename. But not able to use column names inside expression. Please correct me If I am wrong.

While writing expressions for file names we cannot use column names there. We can use parameters there. So we should have our desired values in to parameters and use them in to file names.

Please check below screen shots which gives better idea.

Hence, please consider having your year and month values in to dataflow parameters.

Also, you may consider using cache lookups. Please check this video to understand cache lookups and cache sinks and see if that helps. Thank you.

Hope this helps. Please let us know if any further queries.

----------------

Please consider hitting Accept Answer button. Accepted answers help community as well.