Data Factory Pipeline, I want to Publish my trigger that I have created through a Http request, is this at all possible.

Clinton Morgan 20

Hello,
I have created a Data Factory Pipeline with help from the below links using a HTTP Rest API call.
https://learn.microsoft.com/en-us/rest/api/datafactory/pipelines/create-or-update?view=rest-datafactory-2018-06-01&tabs=HTTP#code-try-0

https://learn.microsoft.com/en-us/rest/api/datafactory/triggers/create-or-update?view=rest-datafactory-2018-06-01&tabs=HTTP

https://learn.microsoft.com/en-us/rest/api/datafactory/triggers/start?view=rest-datafactory-2018-06-01&tabs=HTTP

I have then Created the Trigger and linked it to the Pipeline. I have then tried to start the Trigger, but keep getting the below error.

{
  "error": {
    "code": "BadRequest",
    "message": "Missing or invalid pipeline references for trigger nameoftrigger",
    "target": "nameoftrigger",
    "details": null
  }
}

If I go into Azure and to my Data Factory, Go to my triggers Start it manually then publish it, I can then manually stop it, go back to my API call and this time it will work.

Is there a way that I don't have to manually go and publish the trigger for the API calls to work.
As I will be creating multiple Pipelines and triggers and don't want to have to log on each time they are created.

Bhargava-MSFT 29,266 Reputation points Microsoft Employee

2024-04-24T18:10:03.2133333+00:00

Hello Clinton Morgan,

Welcome to the Microsoft Q&A forum.

The error occurs when the trigger in the error message is not mapped to any pipeline.

Can you please confirm if the trigger is mapped to the pipeline?
Clinton Morgan 20 Reputation points

2024-04-25T08:34:04.7333333+00:00

Hello Bhargava, Thanks for answering.

Yes I can confirm the trigger is mapped to the pipeline.

When I go into the Data Factory, Manage > Triggers. I can see the trigger I created with the API call.

Name CM_Trigger1 on the right I can see Related and its linked to 1 Status is Stopped.
I can click on the 1 and It says "This trigger is being used by the following".
Pipelines "CM_ReportScheduler1" Which I created from the other API call.

Would it help if I add the Json I sent in the API call when I created the Pipeline and Trigger.
Is there maybe a step I am missing. The Start API call wont work unless I manually start and publish the Trigger I can then stop the trigger manually and publish again so its in a stopped state, Then I can use the Start "Post" API call" please see http Post URL below that will start the trigger. This then works with no errors and the Pipeline works as expected.

"/management.azure.com/subscriptions/ID/resourceGroups/ID/providers/Microsoft.DataFactory/factories/DataFactoryName/triggers/CM_Trigger1/start?api-version=2018-06-01"

Clinton Morgan 20

Json for the Creation of the Pipline
URL for the PUT request
"/management.azure.com/subscriptions/ID/resourceGroups/ID/providers/Microsoft.DataFactory/factories/DataFactoryName/pipelines/CM_ReportScheduler1?api-version=2018-06-01"
Body

{
    "name": "CM_ReportScheduler1",
    "type": "Microsoft.DataFactory/factories/pipelines",
    "properties": {
        "description": "Pipeline to run Tonnage Report 2",
        "activities": [
            {
                "name": "Run Stored Procedure Create Job",
                "type": "SqlServerStoredProcedure",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 3,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "storedProcedureName": "[dbo].[CreateJob]",
                    "storedProcedureParameters": {
                        "jobId": {
                            "value": {
                                "value": "@pipeline().parameters.JobId",
                                "type": "Expression"
                            },
                            "type": "Int32"
                        }
                    }
                },
                "linkedServiceName": {
                    "referenceName": "AzureScheduleDataFactoryLink",
                    "type": "LinkedServiceReference"
                }
            }
        ],
         "parameters": {
            "JobId": {
                "type": "int",
                "defaultValue": 30691
            }
        },
        "annotations": [],
        "lastPublishTime": "2024-04-22T13:48:59Z"
    }
}

Clinton Morgan 20

Json for the creation of the trigger
URL for the PUT request
"/management.azure.com/subscriptions/ID/resourceGroups/ID/providers/Microsoft.DataFactory/factories/DataFactoryName/triggers/CM_Trigger1?api-version=2018-06-01"
Body



{
    "name": "CM_Trigger1",
    "properties": {
        "type": "ScheduleTrigger",
        "typeProperties": {
            "recurrence": {
                "frequency": "Minute",
                "interval": 10,
                "startTime": "2024-04-23T11:00:00.8441801Z",
                "timeZone": "GMT Standard Time"
            }
        }, 
        
        "pipelines": [
            {
                "pipelineReference": {
                "referenceName": "CM_ReportScheduler1"
                },
                "parameters": {}
            }
        ],
    "annotations": [],
    "description": "Publishing trigger changes"

    }
}

Bhargava-MSFT 29,266 Reputation points Microsoft Employee

2024-04-26T20:04:26.6333333+00:00

Hello Clinton ,

I checked with my internal team and here is the suggestion.

first try to create trigger using API without providing any pipeline reference (https://learn.microsoft.com/en-us/rest/api/datafactory/triggers/create-or-update?view=rest-datafactory-2018-06-01&tabs=HTTP)
and after creation is successful, then start trigger using API if it is started then update the same trigger by adding pipeline reference then start the trigger using API again(https://learn.microsoft.com/en-us/rest/api/datafactory/triggers/start?view=rest-datafactory-2018-06-01&tabs=HTTP)

Please try the above and let me know.
Clinton Morgan 20 Reputation points

2024-04-29T09:45:54.1133333+00:00
Hi BhargavaGunnam-MSFT, thanks for getting back to me.
I followed your suggestions and still no luck.
Created the Trigger and tried to start it, but came back with the following error.

{ "error": { "code": "BadRequest", "message": "Missing or invalid pipeline references for trigger CM_Trigger2", "target": "CM_Trigger2", "details": null } }

I can see the trigger on Azure and its in a stopped status,
If I manually start then publish, manually stop and publish it still has a problem, If I then run the API with a reference to the pipeline it still wont work, But If I then manually start publish then manually stop then publish with the trigger linked to the pipeline, then run the API call to start the trigger it works.

It seems it wants it to be published manually before the API start will have an effect.

Is it maybe some permissions that I don't know about, that will let the API Start have the ability to publish the trigger and start it in the same call?

Thank once again for the help Bhargava.
Bhargava-MSFT 29,266 Reputation points Microsoft Employee

2024-04-29T19:52:49+00:00

Hello Clinton,

Thanks for the additional details. This seems to be a bug or a permissions issue.

I suggest filing a support request so that a support engineer can further troubleshoot the issue during a screen-sharing session. If you don't have a support plan, please let me know. I can provide a one-time free support request to work on this case.

I look forward to hearing from you.
Clinton Morgan 20 Reputation points

2024-05-09T08:48:56.3833333+00:00

Hi Bhargava

Sorry for not responding sooner. Please can I ask if you can organize a one-time free support request. What do I need to do to proceed. Thanks once again for all your assistance.
Bhargava-MSFT 29,266 Reputation points Microsoft Employee

2024-05-09T16:29:32.2333333+00:00

Hello Clinton,

No worries :)

I have sent you a private message. Could you please check it and provide the requested details so we can enable a one-time free support request on your subscription?

Accepted answer

Eduardo Palomo Rodriguez 80 Reputation points Microsoft Employee

2024-07-08T07:12:31.58+00:00

Hello all, this problem occurs because the trigger body is not complete when you create the trigger via API call, the pipelineReference block lacks the field "type": "PipelineReferece". If you compare with the official documentation, you will see it there:
https://learn.microsoft.com/en-us/rest/api/datafactory/triggers/create-or-update?view=rest-datafactory-2018-06-01&tabs=HTTP#triggers_create

Your body:

Documentation:

The UI will auto-fix the code to ensure consistency but the fixed code is not applied till the user Publish (deploy the changes internally).

To solve this problem, you only need to add that "type": "PipelineReferece" in your trigger body when creating the trigger via API, and you will be able to start it afterwards.
Please sign in to rate this answer.

1 person found this answer helpful.
Clinton Morgan 20 Reputation points

2024-07-08T09:50:55.3733333+00:00

Thank you Bhargava and Pedro for getting back to me and finding the problem with my problem.
Sign in to comment

Share via

Data Factory Pipeline, I want to Publish my trigger that I have created through a Http request, is this at all possible.

0 additional answers