Dataflow fails when executing pipeline and presents the following error

Question

Dataflow fails when executing pipeline and presents the following error

Eddynson Vega 152

{"message":"Job failed due to reason: The store configuration is not defined. This error is potentially caused by invalid parameter assignment in the pipeline.. Details:","failureType":"UserError","target":"Data flow","errorCode":"DF-Executor-StoreIsNotDefined"}

When I debug the dataflow it works fine,but when I execute it though a trigger it presents the error.

ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2021-10-04T06:06:31.933+00:00
Hi @Anonymous ,

Thank you for posting query on Microsoft Q&A Platform.

It seems you configuration inside your dataflows are parameterized and when you are debugging you might be passing that configurations information correctly in to that parameters.

But when you are triggering pipeline, you may be passing the values in to these parameters runtime by fetching data from somewhere may be. I feel your runtime values in to you parameters are incorrect and hence you are ending up with this issue.

Could you please review your values what you are passing in to dataflow runtime and see if that causing issue?

Could you please share more details on your dataflow implementation step by step so that we can try to repro scenario and see? Thank you
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2021-10-05T09:20:02.117+00:00

Hi @Anonymous ,

Following up to see if you get chance to look above comment and share details?
Eddynson Vega 152 Reputation points

2021-10-05T15:49:28.683+00:00

Hi @ShaikMaheer-MSFT ,

In the execution trigger I have carefully reviewed the dynamically defined parameter of an activity of type Lookup and sends it as a string. Example:
Eddynson Vega 152 Reputation points

2021-10-05T15:51:40.157+00:00

I use this parameter in the dataflow filter, to filter the data that have dates greater than the date of the parameters.
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2021-10-05T17:19:57.513+00:00

Hi @Anonymous ,

Thank you for sharing details. So, the value which are passing runtime is matching with the format of the value which you are using during your debug dataflow. I guess that might some difference between the format of value between runtime and debug. You might need to format it accordingly to correct your error.

Could you please share your debug dataflow details. Like what is your parameter type? and what value you are passing to it during debug data preview? and how you are using that parameter in filter transformation expression?
Eddynson Vega 152 Reputation points

2021-10-05T19:59:02.337+00:00

@ShaikMaheer-MSFT

The date value that is being sent at run time does not match the format of the value at data processing time.

- Date format that is sent at run time: 2020-01-26T00:00:00Z

- Date format in data processing: 2020-02-27

Note: Something important to note is that both types of data are different, the one that is sent dynamically is of type string, and the one that is compared in the processing is of type date.

That is why in the filter step there is a date type convesion for the parameter that is received in string and that is why you have a split expression to make a cut in the letter T and in this way achieve that both formats coincide, but I do not know if it is really okay.

Example:

Accepted answer

0 additional answers

Your answer

ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2021-10-04T06:06:31.933+00:00

Hi @Anonymous ,

Thank you for posting query on Microsoft Q&A Platform.

It seems you configuration inside your dataflows are parameterized and when you are debugging you might be passing that configurations information correctly in to that parameters.

But when you are triggering pipeline, you may be passing the values in to these parameters runtime by fetching data from somewhere may be. I feel your runtime values in to you parameters are incorrect and hence you are ending up with this issue.

Could you please review your values what you are passing in to dataflow runtime and see if that causing issue?

Could you please share more details on your dataflow implementation step by step so that we can try to repro scenario and see? Thank you
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2021-10-05T09:20:02.117+00:00

Hi @Anonymous ,

Following up to see if you get chance to look above comment and share details?
Eddynson Vega 152 Reputation points

2021-10-05T15:49:28.683+00:00

Hi @ShaikMaheer-MSFT ,

In the execution trigger I have carefully reviewed the dynamically defined parameter of an activity of type Lookup and sends it as a string. Example:
Eddynson Vega 152 Reputation points

2021-10-05T15:51:40.157+00:00

I use this parameter in the dataflow filter, to filter the data that have dates greater than the date of the parameters.
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2021-10-05T17:19:57.513+00:00

Hi @Anonymous ,

Thank you for sharing details. So, the value which are passing runtime is matching with the format of the value which you are using during your debug dataflow. I guess that might some difference between the format of value between runtime and debug. You might need to format it accordingly to correct your error.

Could you please share your debug dataflow details. Like what is your parameter type? and what value you are passing to it during debug data preview? and how you are using that parameter in filter transformation expression?
Eddynson Vega 152 Reputation points

2021-10-05T19:59:02.337+00:00

@ShaikMaheer-MSFT

The date value that is being sent at run time does not match the format of the value at data processing time.

- Date format that is sent at run time: 2020-01-26T00:00:00Z

- Date format in data processing: 2020-02-27

Note: Something important to note is that both types of data are different, the one that is sent dynamically is of type string, and the one that is compared in the processing is of type date.

That is why in the filter step there is a date type convesion for the parameter that is received in string and that is why you have a split expression to make a cut in the letter T and in this way achieve that both formats coincide, but I do not know if it is really okay.

Example:

Answer 1

Hi @Anonymous ,

Thank you for Posting query on Microsoft Q&A and sharing few insights on issue.

I tried similar implementation. by having a parameter in data flow and supplying string value to that parameter from data flow activity. "

I see error when I check Expression box inside Dataflow Parameters tab. When I Uncheck Expression box then I am not seeing any Issue.

When you uncheck expression box that means the string parameter values will be wrapped into single quotes using the interpolation syntax

You try to execute your data flow by checking and un-checking expression box to see how it behaves.

Below are screenshots:

How ever, Your error message in the question still makes little confusion as it says store not defined. That means you may be using some dataset which may be pointing to a storage which is not defined. Kindly review this point as also once. Thank you.

Hope this will help. Please let us know how it goes and if any further queries.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Eddynson Vega 152 Reputation points

2021-10-07T01:38:08.037+00:00

@ShaikMaheer-MSFT Thank you, this has solved my problem.
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2021-10-07T04:15:09.18+00:00

@Anonymous - Happy to know that your issue resolved. Thank you.
Priyaroop Singh 1 Reputation point

2023-04-02T23:40:54.0666667+00:00

This was driving me crazy. I noticed that the pipelines that fail were passing the date string parameter without the single quotes. Your solution worked like a charm.

Thanks

Share via

Dataflow fails when executing pipeline and presents the following error

0 additional answers

Your answer