Azure Data Factory: The pipeline does not start with Trigger now , but same pipeline starts and runs with Debug

Sunita Gannavarapu 66 Microsoft Employee

Hi Team, I am facing issue. The pipeline started when I used debug and ran fine. But when I use trigger now to run the pipeline it fails saying

{"code":"BadRequest","message":"Invalid value for property 'parameters'","target":"pipeline/ExportCSVPipeline/runid/771c435e-26eb-47c6-89d0-8f28e361b47a","details":null

I provided the same value for the parameter when running via Debug.

did a publish and tested but same result. Verified the Json(input parameter value) and its valid Json.

This issue is similar to the issue mentioned in : https://learn.microsoft.com/en-us/answers/questions/107357/azure-data-factory-trigger-is-fired-but-pipeline-d?page=1&orderby=Helpful&comment=answer-119031#newest-answer-comment

Can you elaborate what action was taken to solve the issue ?

Suba Balaji 11,206 Reputation points

2023-10-06T12:23:13.46+00:00

Can you please post the parameters and its value you provided for trigger?

Sunita Gannavarapu 66 Microsoft Employee

Hi Suba, Than you for your response , Pls find the json used as the parameter:

{
    "IntegrationJobLogId": 2158506,
    "IntegrationJobId": 94,
    "JobCode": "ADFEXPORT",
    "Token": "77037be0-f8a8-4463-97a0-2c9e7a8fa020",
    "JobGroupLogToken": null,
    "StartDateTime": "2023-10-06T06:33:25.6894269Z",
    "EndDateTime": null,
    "TotalRequestCount": 1,
    "ExpectedEndDateTime": "2023-10-06T06:48:25.6894272Z",
    "Retry": 0,
    "IsScheduled": false,
    "IntegrationJobStatusId": 2,
    "StatusCode": "PendingToBePicked",
    "StatusDisplayText": "Pending To Be Picked",
    "Metadata": {
        "FileName": null,
        "TotalRecords": 0,
        "TotalPassedRecords": 0,
        "JobType": "ADFEXPORT",
        "FileProcessingDate": null,
        "CallbackUrl": null,
        "DurableUrl": null,
        "LastCaregiverSync": null,
        "Path": null,
        "UserEmailId": "swapnilwagh@microsoft.com",
        "JobData": "{\"jobType\":\"ADFEXPORT\",\"batchStatus\":\"In Progress\",\"companyId\":\"3\",\"sqlServer\":\"u2-sql-qa\",\"databaseName\":\"u2billing_cdwa\",\"storedProcName\":[\"exec billing.usp_GetPaymentBatchHeaderDetails 'PBCN2023278SHF0000001','3',NULL\",\"exec billing.usp_GetPaymentBatchCcnLcnDetails 'PBCN2023278SHF0000001','3',NULL\",\"exec billing.usp_GetPaymentBatchUnappliedPayDetails 'PBCN2023278SHF0000001','3',NULL\",\"exec billing.usp_GetPaymentBatchMCNDetails 'PBCN2023278SHF0000001','3',NULL\"],\"fileName\":[\"2023-10-06/PaymentBatch-20231006T063225454-BatchHeader.csv\",\"2023-10-06/PaymentBatch-20231006T063225454-AppliedPayment.csv\",\"2023-10-06/PaymentBatch-20231006T063225454-UnappliedPayment.csv\",\"2023-10-06/PaymentBatch-20231006T063225454-Manual.csv\"],\"zipFileName\":\"2023-10-06/PaymentBatch-20231006T063225454.zip\",\"emailSubject\":\"File for Payment Batch is ready for download\",\"guid\":\"5798a90d-c46a-4803-a3f4-2fbb77cc3bdf\",\"sendEmailUrl\":\"https://shared-servicebus-qa.servicebus.windows.net/emailsend/messages\",\"dynamicPartition\":null}",
        "StagedBlobPath": null
    },
    "ConfigMetadata": null,
    "IsSuccessRequired": false,
    "CompanyCode": "CDWA",
    "TimeExtension": 0,
    "ExpectedJobDuration": 15,
    "IsTerminal": false,
    "JobSubCategoryCode": "EXPORT",
    "DestinationType": "ServiceBus"
}

has context menu

Suba Balaji 11,206 Reputation points

2023-10-09T04:55:53.37+00:00

Hi
Sunita Gannavarapu

What are you trying to do with the above param value? Like what transformations\ activities you use in your pipeline by passing this param value? Please explain.

I simply tried to use a set variable activity and passed on this json from parameter. the pipeline works fine in debug as well as manual trigger.

So want to know what other activities you use in PL so that we can figure out the issue.

Awaiting your response.

Thanks
Sunita Gannavarapu 66 Reputation points Microsoft Employee

2023-10-09T08:00:57.18+00:00

I Am using them for set other variable and then run a stored procedure and other web activity using this variables. it is working for me in debug but not on trigger mode.

this pipeline was earlier running fine with set triggers as well.
Suba Balaji 11,206 Reputation points

2023-10-09T16:11:22.0566667+00:00

Could you please post the expressions used in set variable and settings or sp and web activity. This kind of error usually occurs when we have some syntax issues in activities.
Sunita Gannavarapu 66 Reputation points Microsoft Employee

2023-10-10T05:26:49.93+00:00

I do not understand why we should have issue with set variable activity, when it is working with same input in debug mode. With trigger now , it is not even going to set variable activity.
Suba Balaji 11,206 Reputation points

2023-10-10T10:11:11.6633333+00:00

Hi, it could be due to any expression that is being used in the pipeline or the value being assigned to trigger. Try changing the param value to some simple value and see if Pipeline gets triggered.
Sunita Gannavarapu 66 Reputation points Microsoft Employee

2023-10-10T10:22:06.0733333+00:00

Hi Suba , thank for replying back, tried that as well. Ran it with basic json

{"Name":"Swapnil"}

still trigger not working.

Published the pipeline and tried again. Same result
Suba Balaji 11,206 Reputation points

2023-10-11T06:26:37.59+00:00

Sunita Gannavarapu Hi Sunita, I tried to replicate the issue, but unfortunately, the pipeline works fine with similar parameter value and trigger. I suspect something with pipeline design. would be great to if you could post the activities and expressions used for deeper investigation.
KranthiPakala-MSFT 46,442 Reputation points Microsoft Employee

2023-10-17T00:42:47.77+00:00

@Sunita Gannavarapu As @Suba Balaji mentioned we are unable to replicate the issue on our end. If you are still blocked because of this issue, please feel free to reach out to me over team as you are internal so that I can get on a call to check your pipeline configuration to figure out the root cause.

But in the past, I have noticed such issues when customers have incorrect configuration for pipeline parameters or variables.
Sunita Gannavarapu 66 Reputation points Microsoft Employee

2023-10-31T04:40:08.2666667+00:00

@KranthiPakala-MSFT : Thank you for reaching out. The service engineer helped to fix the issue.

Looks like Team made change in 1 of the linked service paramter. They changed it from hardcoded to a dynamic parameter receivable from data source. While making it dynamic, the value was specified in the ARM(JSON) but parameter name was not. This got deployed. and then it started breaking. Not a deployed issue though more of a coding error.

Thank you.

Accepted answer

KranthiPakala-MSFT 46,442 Reputation points Microsoft Employee

2023-11-01T16:49:45.42+00:00
@Sunita Gannavarapu I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others Opens in new window or tab", I'll repost your solution in case you'd like to accept the answer Opens in new window or tab.

Error Message:

{"code":"BadRequest","message":"Invalid value for property 'parameters'","target":"pipeline/ExportCSVPipeline/runid/771c435e-26eb-47c6-89d0-8f28e361b47a","details":null

Issue:

The pipeline does not start with Trigger now , but same pipeline starts and runs with Debug

Solution:

Looks like Team made change in 1 of the linked service paramter. They changed it from hardcoded to a dynamic parameter receivable from data source. While making it dynamic, the value was specified in the ARM(JSON) but parameter name was not. This got deployed. and then it started breaking. Not a deployed issue though more of a coding error.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

I hope this helps!

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment

Share via

Azure Data Factory: The pipeline does not start with Trigger now , but same pipeline starts and runs with Debug

0 additional answers