Inconsistent behaviour with Data Factory Expressions

Question

Why is there an is inconsistency between how expressions behave in the copy activity when using different data source types?

Teradata:
When I have nested json as a parameter on a Teradata ds and I make the param PAR_CONFIG of type object, I get an error when trying to use the following expression:

Failing Expression

@pipeline().parameters.PAR_CONFIG.source.datasetProperties.AKVSecret  

Error:

The parameters and expression cannot be resolved for schema operations. Error Message: {
"message": "ErrorCode=InvalidTemplate, ErrorMessage=The expression 'pipeline().parameters.PAR_CONFIG.source.datasetProperties.AKVSecret' cannot be evaluated because property 'source' cannot be selected. Property selection is not supported on values of type 'String'

To get it to work I need to change the expression to:

@json(string(pipeline().parameters.PAR_CONFIG)).source.datasetProperties.AKVSecret  

Oracle:
When I have nested json as a parameter on an Oracle ds and I make the param PAR_CONFIG of type object, the expression works that failed for Teradata:

Working Expression:

@pipeline().parameters.PAR_CONFIG.source.datasetProperties.AKVSecret  

The expression which works on Teradata however does not work for the Oracle ds:

@json(string(pipeline().parameters.PAR_CONFIG)).source.datasetProperties.AKVSecret  

The same json stucture is being passed to the Oracle and the Teradata DS, with only the assigned property values being different. Its behavioral inconsistencies like this that makes developing pipeline's in ADF incredibly frustrating.

Answer 1

Hi @Waylon ,

Thank you for posting query in Microsoft Q&A Platform.

error:
The parameters and expression cannot be resolved for schema operations. Error Message: {
"message": "ErrorCode=InvalidTemplate, ErrorMessage=The expression 'pipeline().parameters.PAR_CONFIG.source.datasetProperties.AKVSecret' cannot be evaluated because property 'source' cannot be selected. Property selection is not supported on values of type 'String'

I was able to repro above error in the case when we declared parameter as string type but passing json object as value to it or when we are passing json data as string not as object into object type of parameter.

I believe either of these cases might be root cause in your case. Kindly check parameter types and data types which we are passing into parameter.

Hope this helps. If not, then kindly share more details on implementation where you are passing values to parameters along with parameter types in screenshots. That helps to understand your specific case better.

-------------------

Please consider hitting Accept Answer button. Accepted answers help community as well.