Azure Data Factory: Job failed due to reason Expression cannot be parsed

Question

Azure Data Factory: Job failed due to reason Expression cannot be parsed

Kyle Maysey 21

I am attempting to use a pipeline parameter in a ForEach activity, however I am receiving a "DF-Executor-ParseError" error with no details:

In checking the activities within the ForEach we can see the incoming parameters evaluate as expected:

And if I plug in the values directly into the Debug Settings the underlying flow runs just fine:

I've attached copies of the Pipeline definition as well as the Data Flow script. Does anyone have an idea on what I might be missing?

Thank you!

114705-pipeline-manually-run-items.txt
114717-data-flow-script-test-generics.txt

Ryan Abbey 1,186 Reputation points

2021-07-14T20:28:39.207+00:00

Afraid I haven't used the regex component but are you sure you need the single quotes or the escaping of the backslash? I would expect the field to take the regex string and handle any escaping if required
Kyle Maysey 21 Reputation points

2021-07-16T01:10:33.807+00:00

Thank you Ryan. In testing the Data Flow directly, the single quotes and escaping does appear to be required by ADF. The value of the field is evaluated by the expression builder which need the quotes for identifying strings.

The string without the escaping does return a rather familiar error message.

An "Unable to Parse" with no details.

In checking if the single quotes are causing problems in the Pipeline itself, I can check the Expression box for the parameter on the Data Flow call.

This removes the single quotes when the Pipeline passes the parameter down.

However the same error is returned.

Thank you for the suggestion though!
Ryan Abbey 1,186 Reputation points

2021-07-16T03:50:34.59+00:00

Sorry I couldn't be more help

What I would suggest is start small, rather than passing a variable, pass a small regex string and validate that it results in what you want then gradually build up to your output

One observation I do make, in your screenshot showing "no single quotes", I wouldn't expect the double backslash here too for a standard regex string, a double backslash is effectively saying this is a literal backslash whereas you want to say the \d represents a number
Kyle Maysey 21 Reputation points

2021-07-16T16:37:20.097+00:00

I had already tried sending a literal regex string in my initial testing as a proof-of-concept and that worked as intended. I provided additional details in response to Shaik's answer, but it's only when I attempt to use the variable that contains the string that it starts having a problem. This is true whether I use the iterator directly, or read the value into a separate pipeline variable and pass it down.

As for he extra backslashes, that seems to be added by the ADF platform itself. The regex patterns still work as intended as it revaluates the escaping before sending through the regex procedure, so the \d becomes \d at the regex runtime. I should also clarify that the pattern being fed into ADF only contains a single backslash.

Thank you again for taking a look!
Ryan Abbey 1,186 Reputation points

2021-07-18T20:20:18.83+00:00

Hmm, ok, hopefully you get the solution soon, I'll be keen to know what it was! Good luck!

Accepted answer

0 additional answers

Your answer

Ryan Abbey 1,186 Reputation points

2021-07-14T20:28:39.207+00:00

Afraid I haven't used the regex component but are you sure you need the single quotes or the escaping of the backslash? I would expect the field to take the regex string and handle any escaping if required
Kyle Maysey 21 Reputation points

2021-07-16T01:10:33.807+00:00

Thank you Ryan. In testing the Data Flow directly, the single quotes and escaping does appear to be required by ADF. The value of the field is evaluated by the expression builder which need the quotes for identifying strings.

The string without the escaping does return a rather familiar error message.

An "Unable to Parse" with no details.

In checking if the single quotes are causing problems in the Pipeline itself, I can check the Expression box for the parameter on the Data Flow call.

This removes the single quotes when the Pipeline passes the parameter down.

However the same error is returned.

Thank you for the suggestion though!
Ryan Abbey 1,186 Reputation points

2021-07-16T03:50:34.59+00:00

Sorry I couldn't be more help

What I would suggest is start small, rather than passing a variable, pass a small regex string and validate that it results in what you want then gradually build up to your output

One observation I do make, in your screenshot showing "no single quotes", I wouldn't expect the double backslash here too for a standard regex string, a double backslash is effectively saying this is a literal backslash whereas you want to say the \d represents a number
Kyle Maysey 21 Reputation points

2021-07-16T16:37:20.097+00:00

I had already tried sending a literal regex string in my initial testing as a proof-of-concept and that worked as intended. I provided additional details in response to Shaik's answer, but it's only when I attempt to use the variable that contains the string that it starts having a problem. This is true whether I use the iterator directly, or read the value into a separate pipeline variable and pass it down.

As for he extra backslashes, that seems to be added by the ADF platform itself. The regex patterns still work as intended as it revaluates the escaping before sending through the regex procedure, so the \d becomes \d at the regex runtime. I should also clarify that the pattern being fed into ADF only contains a single backslash.

Thank you again for taking a look!
Ryan Abbey 1,186 Reputation points

2021-07-18T20:20:18.83+00:00

Hmm, ok, hopefully you get the solution soon, I'll be keen to know what it was! Good luck!

Answer 1

ShaikMaheer-MSFT 38,546 Microsoft Employee Moderator

Hi @Kyle Maysey ,

Welcome to Microsoft Q&A Platform. Thank you for posting your query here.

I tried to repro your scenario by creating parameters and passing similar values in it. I haven't got any issues too. Please check below gifs for same.

Data flow Activity:

Data Flow Execution:

Hence, It seems values what you are passing from your expressions in to parameters may some where causing data flow to break. It would be great if you can check the values what your execution is passing in to parameters.

Kindly feel free to share the exact values which causing your execution to break during run time. So, that we can help you to fix that as well. Thank you.

------------------------------------------------

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
Want a reminder to come back and check responses? Here is how to subscribe to a notification.

Kyle Maysey 21 Reputation points

2021-07-16T00:53:54.46+00:00

Thank you for your response. Here are more details about the Pipeline and the values it passes down to the Data Flow:

The Pipeline starts with a lookup to a table that stores the Regular Expressions.

The row of the table contains the pattern, the target table, the target column, and the name/label for the data being returned by the pattern.

The Regular Expressions are fed into a ForEach activity, referencing the results of the Lookup.

IThe activity in the ForEach calls the Data Flow and provides the parameters.

Currently I feed the iterator value through a variable.

I've tried using @item().Pattern directly in the DF call, but got the same error.

I've attached the input JSON for one iteration of the loop. Is there any other information I can provide?

Thank you again!

115241-adf-foreach-input-json.txt
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2021-07-20T00:55:32.413+00:00

Hi @Kyle Maysey ,

We reached product team internally on this issue. We will share updates soon once we see response from product team. Thanks for being patience.
Mithun Kumar Rajendran 6 Reputation points Microsoft Employee

2021-07-21T01:43:08.297+00:00
@Kyle Maysey - This is because dataflow requires regex to be escaped like \s+ needs to be \s+.

Dataflow's expression editor will force you to do this like for example:

ORDER OF:\s+\d+(.?).AP would need to passed in as ORDER OF:\s+\d+(.?).AP

It appears that the Lookup you are using has the pattern without escaping as this 'ORDER OF:\s+\d+(.*?).AP'
That's why it is failing when passed into dataflow.

Two ways to fix this :

Adding an escape slash like above in the table you are looking up from so that the column value is ORDER OF:\s+\d+(.*?).AP?

Updating the Set Variable value to @replace(item().Pattern,'\','\') so that slash is escaped properly
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2021-07-22T05:44:44.087+00:00

Hi @Kyle Maysey ,

Glad to know that your issue resolved. Thank you.

Share via

Azure Data Factory: Job failed due to reason Expression cannot be parsed

0 additional answers

Your answer