Data flow activity is not dynamically detecting new projections in source

Question

Dear Microsoft,

We are currently developing a data quality (DQ) solution using Azure Data Factory. The ADF pipeline should ultimately read DQ expressions from an Azure SQL table and run these expressions against pre-defined ADLS files. As an outcome, the solution should generate DQ results for each ADLS file.

Our ADF pipeline has been highly successful at fetching the DQ expressions and ADLS file names from the Azure SQL table, but we are facing some issues when trying to dynamically pass these input parameters to a downstream data flow.

More specifically, we are seeing the following message in the error: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: Column name used in expression is unavailable or invalid","Details"

It appears as if the source of our data flow is struggling to detect the projection of new ADLS files that we are dynamically passing to the data flow. If we manually import the new projection, the data flow runs successfully.

We've done some investigation and the following thread related to a very similar problem: https://learn.microsoft.com/en-us/answers/questions/152133/azure-data-factory-workflow-stopped-working-in-aut.html

Do you have any advice for the above problem? We are trying to avoid manual intervention as we will have to run this solution on a regular basis and the solution should be able to successfully process the addition of new entries (i.e. DQ expressions) in the Azure SQL table.

Answer 1

Hi @Anonymous ,

Thank you for posting query in Microsoft Q&A Platform.

As per my understanding, your data flow expression referring to some column which is missing and resulting in error. Correct me if my understanding wrong.

This error may happening because of multiple reasons. Kindly make sure all above pointers in your case and see if that helps.

• Is your source transformation datasets has schema in them? If yes, kindly clear it as your intention is to have dynamic schema.
• Kindly make sure the columns which your expression referring is actually exists in source.
• Under source transformation make sure to have "Allow Schema Drift" & "Infer drifted column data types" selected. Make sure to unselect "Validate schema" option.
• If you are using auto column mapping in sink transformation make sure to have source and sink column names matching.

Try to perform "Data Preview" and see if its working. If yes, kindly do some minor change and publish your changes back.

Hope this helps. Please let us know how it goes.

-------------

Please consider hitting Accept Answer. Accepted answers helps community as well.