Getting error "{"code":"BadRequest","message":null,"target":"pipeline...while debugging a DataFlow within a For-each loop .

Question

Getting error "{"code":"BadRequest","message":null,"target":"pipeline...while debugging a DataFlow within a For-each loop .

Niladitya Das 0

While running Data Flow from a foreach loop the parameter of data sets are not being passed( strange enough, only for the SYNC). Tried creating a pipeline with the same DF and it is successfully running. Again while trying to call the pipeline from loop, the error returns. There could be a question why DF is being called from inside a for loop, the reason is we need multiple JSON output from multiple CSV sources.

Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2024-02-08T20:04:48.9833333+00:00

Hello Niladitya Das

Please correct me if my understanding is wrong. You are encountering an issue with passing parameters to Data Flow activities when running them from within a For Each loop in ADF.

You can add a parameter in Data Flow and set the Data Flow parameter with a Pipeline expression. Then you can set the ForEach item() as the Data Flow parameter. This will allow you to use the item() from the ForEach loop in the Data Flow and fetch that record from the CSV file.

Please see this stackoverflow for your reference: https://stackoverflow.com/questions/61623462/using-mapping-data-flows-within-for-each-activity-of-azure-data-factory-pipeline

I hope this helps.
Niladitya Das 0 Reputation points

2024-02-09T05:02:32.2866667+00:00

Hello @Bhargava-MSFT No ... we are getting this issue while passing a dataset parameters which is set in Linked Service , Datasets level ... the sink is not taking the parameter which is passed in pipeline level
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2024-02-14T21:33:05.13+00:00

Hello Niladitya Das, if the sink is not taking the parameter that is passed in the pipeline, it iseems like the parameter is not being passed correctly or that the sink is not configured to use the parameter. It's difficult to troubleshoot the issue without looking at the pipeline. If you are unable to resolve the issue on your own, I would suggest opening a support case so that a support engineer can further investigate the issue in a screen sharing session. If you do not have a support plan, please let me know so I can provide a one-time support request to work on this case.