Data flow expression builder syntax error with parameters

Question

I have a dataflow that works fine when values are static. However if I try to use Parameters, I get the following error

Spark job failed: { "text/plain": "{"runId":"ada2e29b-79bc-4ec3-afe0-0ec32b4bcf91","sessionId":"2990dd25-b71b-44f4-901a-90e2adebdede","status":"Failed","payload":{"statusCode":400,"shortMessage":"DF-EXP-061 at Source 'source1'(Line 2/Col 11): Column operands are not allowed in literal expressions. Possible causes being value for a string parameter or an expected string value is not enclosed in single quotes. Please check for the same near the mentioned line number(s) in the data flow script.","detailedMessage":"Failure 2024-09-02 07:16:35.594 failed DebugManager.processJob, run=ada2e29b-79bc-4ec3-afe0-0ec32b4bcf91, errorMessage=DF-EXP-061 at Source 'source1'(Line 2/Col 11): Column operands are not allowed in literal expressions. Possible causes being value for a string parameter or an expected string value is not enclosed in single quotes. Please check for the same near the mentioned line number(s) in the data flow script."}}\n" } - RunId: ada2e29b-79bc-4ec3-afe0-0ec32b4bcf91

In my source options, If I use 'Bearer ' + $AuthString I receive the above error, and the auto-generated scriptview looks like this

headers: ['authorization' -> ('Bearer ' + $AuthString)],

However, if I use 'Bearer ' + '$AuthString' I don't get the syntax error, but the http request fails due to 401 unauthorised
The scriptview then looks like this

headers: ['authorization' -> ('Bearer ' + '$AuthString')],

The parameter $AuthString is of type string

Answer 1

The key issue here is the difference between treating a parameter as a string literal versus a column reference or expression.

Explanation:

• 'Bearer ' + $AuthString:
  • In this case, $AuthString is treated as a column reference or expression, not as a literal value. The error suggests that column operands are not allowed in literal expressions, which indicates that the system is expecting a literal string but is instead interpreting $AuthString as something else.
• 'Bearer ' + '$AuthString':
  • Here, '$AuthString' is treated as a literal string, meaning the literal string Bearer $AuthString will be used in the header. This is why you don’t get a syntax error, but the resulting HTTP request fails with a 401 unauthorized error, as the literal string doesn’t represent a valid authorization token.

Solution:

To properly concatenate the string literal 'Bearer ' with the parameter $AuthString, you should ensure that the parameter is being correctly interpolated as a string. This can be achieved by using the string() function to explicitly convert the parameter to a string within the expression.

Correct Expression:

headers: ['authorization' -> ('Bearer ' + string($AuthString))],