Getting error "{"code":"BadRequest","message":null,"target":"pipeline..." when trying to debug pipeline

Anonymous

Have confirmed through reproducing a couple of times. I have 3 data factories, dev/uat/prod. I develop something on DEV, then use Azure devops to run through and release the ARM templates onto UAT and prod. This all works OK.

However I then change the prod KeyVault Linked Service's default value to something else, using the UI. This is used in other Linked Services to obtain login information.

After changing the linked service "Default value" parameter only and publishing, I then get this error trying to debug a pipeline:

{"code":"BadRequest","message":null,"target":"pipeline//runid/XXX Run ID XXX","details":null,"error":null} (note I removed the GUID run ID in the code)

Seems like that should work without issue, might be a bug?

EDIT:

I have tried a simple pipeline which sets some variables, which works fine. However any pipeline that requires pass through to Key Vault doesn't seem to work. This was all working last week, so something seems to have been screwed up.

Saurabh Sharma 23,761 Reputation points Microsoft Employee

2021-03-09T00:17:20.87+00:00

@Anonymous Thanks for using Microsoft Q&A !!

What is the parameter value you are passing. Does it contains any URL's ?
Anonymous

2021-03-09T01:09:36.91+00:00

Yes, it contains the base URL of the key vault. The Key Vault Base URL is passed through to the Linked Service as a global variable defined as a URL string. See the pic:
Anonymous

2021-03-09T19:45:57.687+00:00

@Saurabh Sharma - any updates? Is there currently a problem passing through URLs?
Anonymous

2021-03-09T20:24:37.973+00:00

Note that trigger runs also now are failing for our prod systems, which previously worked too... we are getting errors like:

Triggering Pipeline Run pipeline name failed with Error ErrorCode=InvalidTemplate, ErrorMessage=The expression '{body('Get domain_id based on domain nameComposeRuntimeVariables')?.ETLControlDB_DB_NoTable851b996b306741ed98bf7c3a977f05c0.key_vault_URL}' is not valid: the string character '{' at position '0' is not expected.\".""

Pipelines aren't even starting...
Saurabh Sharma 23,761 Reputation points Microsoft Employee

2021-03-09T23:06:27.147+00:00

@Anonymous Sorry for the delay. Wanted to know more on how you are using the parameters so the ask. I do not think any issues with passing URL's but I will check on this internally and get back to you.
Anonymous

2021-03-10T22:29:37.76+00:00

@Saurabh Sharma

OK, a couple of other things I have noticed which could be causing the issue. Neither problem existed a few weeks ago:

Firstly my KeyVault references on other Linked services now appear as [object Object], instead of referring to the KeyVault Linked Service which I had previously set up.

Secondly, in the KeyVault Linked service, if I change the BaseURL value via the UI, from the parameter value with curly braces around it ("@{param...}" to "@Paramjeet Dahiya ..."), to one without curly braces, the change doesn't stick after saving. However if I change this via the JSON code, the change sticks. This looks to be a UI issue, rather than anything else. It exists on my DEV version (which is where I do dev work, then publish and use DevOps to push through to UAT/PROD environments, as per Microsofts recommendation) too.
Harm Lommers 0 Reputation points

2023-06-01T08:54:09.39+00:00

For future readers (as this apparently is still not fixed):

I experienced the same issue when using a Linked Service for an Azure Blob Storage in which the used AKV url was forwarded with a parameter.

What somehow worked for me is entering random input in the default value. It's not used as I pass the correct value from a pipeline. But when I add this, the pipeline runs fine.

Actually even when you remove the default value afterwards, the pipeline still remains ok...

Still feels quite buggy, but it's a more easy workaround then modifying JSON code.

Accepted answer

Saurabh Sharma 23,761 Reputation points Microsoft Employee

2021-03-17T05:07:19.303+00:00

@Anonymous Products team has identified this as a product bug and they have created an workitem internally and working on to provide a fix for the same. Meanwhile, you can use the workaround you have found to circumvent this issue. Thank you so much for reporting this. Appreciate your patience on the same.

----------

Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.
Please sign in to rate this answer.

3 people found this answer helpful.
Anonymous

2021-03-21T20:33:22.137+00:00

Get back to me when this has been fixed @Saurabh Sharma - I am holding off on a few changes until I hear back.

Thanks,
Brent

Saurabh Sharma 23,761 Reputation points Microsoft Employee

2021-03-22T15:17:09.753+00:00

@Anonymous Sure, I will update the thread as I hear back from the products team. Thanks

Saurabh Sharma 23,761 Reputation points Microsoft Employee

2021-03-29T15:02:51.857+00:00

@Anonymous Our products team require few other details to understand your scenario better. So, could you please help providing the below details over an email to azcommunity[at]microsoft[dot]com with this Q&A thread link and subject as "Attn: Saurabh' and I will share these with the products team ?

Download and share Support Files (using the steps below) as a Zip file

Data Factory Name :
Pipeline Name :
Pipeline Run ID:
Activity Run ID:

Thanks
Saurabh

Darren Cullen 1 Reputation point

2021-04-12T19:58:14.577+00:00

Hi,

Any update on when a fix for this issue is likely to be available?

Also, when you mention "Meanwhile, you can use the workaround you have found to circumvent this issue", what exactly is this workaround?

Thanks

Anonymous

2021-04-12T20:58:53.607+00:00

Hey Darren, the workaround is to modify the json code, not using the UI. You can do this through the UI by going to linked services and hitting the Code button on your KeyVault:

In saying that, they are clearly messing around with the UI as I have sometimes successfully used the UI and sometimes not. It seems to be related to another bug where curly braces are sometimes added around parameters passed through to a linked service, when you open the linked services UI and close it again without updating it. The only way to stop this occurring is to edit the code manually via the button in the pic above.

jhanns syr 1 Reputation point

2023-03-23T15:04:34.5533333+00:00

any updates on this critical severity SIR? it has been over a year since this SIR was declared a product bug. also request an example demonstrating the workaround suggested?

kind regards

Saurabh Sharma • 17,316 Reputation points • Microsoft Employee

Mar 16, 2021, 10:07 PM

@Brent Leslie Products team has identified this as a product bug and they have created an workitem internally and working on to provide a fix for the same.
Sign in to comment

5 additional answers

AM 1 Reputation point

2021-04-22T13:22:21.163+00:00

I wanted to copy multiple tables and convert them to csv (and transfer the csv files into an Azure Storage Account) using Azure Data Factory and looked at this documentation: https://www.modern-dataengineering.com/post/how-to-copy-multiple-tables-in-azure-data-factory

The documentation says in the Sink part:

7. Click source > Query 8. Input the following dynamic sql code: SELECT * FROM @{item().Table_Name}

and below it there is a picture that suggests to use @{item().Table_Name} in the File Name part of File path.

I did as instructed using DelimitedText as the Sink dataset. In there under Connection --> File path --> Filename, I wrote @{item().Table_Name} as instructed. This produced the error {"code":"BadRequest","message":null,"target":"pipeline//runid/...

After we changed @{item().Table_Name} to @{dataset().table_name}.csv the pipeline debugging started to work and the csv files were transferred to the Storage Account with no problems. It seems that the dataset filename must not be hard coded to that field and the @dataset parameter must be used with it.
Please sign in to rate this answer.
Kim Avery (admin) 1 Reputation point

2021-06-24T15:11:17.123+00:00

I'm getting the same error message. My pipeline debugged successfully on yesterday. I Get Metadata first that points to a container with csv files, the ForEach has a data flow activity that includes a Source, Derived Column and Sink. I can do a data preview on all 3. The datasets are all schema less(late binding). I tested the connections successfully. I'm at a lost currently. Any suggestions.

Chantola Kit 0 Reputation points

2023-02-10T15:06:57.8566667+00:00

I had experience with "Failed Connection, Bad RequestId:...". Please make sure that you use MS. Edge or Chrome Internet browser. If you don't please change your browser. It will work.
Sign in to comment

Share via

Getting error "{"code":"BadRequest","message":null,"target":"pipeline..." when trying to debug pipeline

5 additional answers