Data Factory Pipeline, I want to Publish my trigger that I have created through a Http request, is this at all possible.

Clinton Morgan 0

Hello,
I have created a Data Factory Pipeline with help from the below links using a HTTP Rest API call.
https://learn.microsoft.com/en-us/rest/api/datafactory/pipelines/create-or-update?view=rest-datafactory-2018-06-01&tabs=HTTP#code-try-0

https://learn.microsoft.com/en-us/rest/api/datafactory/triggers/create-or-update?view=rest-datafactory-2018-06-01&tabs=HTTP

https://learn.microsoft.com/en-us/rest/api/datafactory/triggers/start?view=rest-datafactory-2018-06-01&tabs=HTTP

I have then Created the Trigger and linked it to the Pipeline. I have then tried to start the Trigger, but keep getting the below error.

{
  "error": {
    "code": "BadRequest",
    "message": "Missing or invalid pipeline references for trigger nameoftrigger",
    "target": "nameoftrigger",
    "details": null
  }
}

If I go into Azure and to my Data Factory, Go to my triggers Start it manually then publish it, I can then manually stop it, go back to my API call and this time it will work.

Is there a way that I don't have to manually go and publish the trigger for the API calls to work.
As I will be creating multiple Pipelines and triggers and don't want to have to log on each time they are created.

BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2024-04-24T18:10:03.2133333+00:00

Hello Clinton Morgan,

Welcome to the Microsoft Q&A forum.

The error occurs when the trigger in the error message is not mapped to any pipeline.

Can you please confirm if the trigger is mapped to the pipeline?
Clinton Morgan 0 Reputation points

2024-04-25T08:34:04.7333333+00:00

Hello Bhargava, Thanks for answering.

Yes I can confirm the trigger is mapped to the pipeline.

When I go into the Data Factory, Manage > Triggers. I can see the trigger I created with the API call.

Name CM_Trigger1 on the right I can see Related and its linked to 1 Status is Stopped.
I can click on the 1 and It says "This trigger is being used by the following".
Pipelines "CM_ReportScheduler1" Which I created from the other API call.

Would it help if I add the Json I sent in the API call when I created the Pipeline and Trigger.
Is there maybe a step I am missing. The Start API call wont work unless I manually start and publish the Trigger I can then stop the trigger manually and publish again so its in a stopped state, Then I can use the Start "Post" API call" please see http Post URL below that will start the trigger. This then works with no errors and the Pipeline works as expected.

"/management.azure.com/subscriptions/ID/resourceGroups/ID/providers/Microsoft.DataFactory/factories/DataFactoryName/triggers/CM_Trigger1/start?api-version=2018-06-01"

Clinton Morgan 0

Json for the Creation of the Pipline
URL for the PUT request
"/management.azure.com/subscriptions/ID/resourceGroups/ID/providers/Microsoft.DataFactory/factories/DataFactoryName/pipelines/CM_ReportScheduler1?api-version=2018-06-01"
Body

{
    "name": "CM_ReportScheduler1",
    "type": "Microsoft.DataFactory/factories/pipelines",
    "properties": {
        "description": "Pipeline to run Tonnage Report 2",
        "activities": [
            {
                "name": "Run Stored Procedure Create Job",
                "type": "SqlServerStoredProcedure",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 3,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "storedProcedureName": "[dbo].[CreateJob]",
                    "storedProcedureParameters": {
                        "jobId": {
                            "value": {
                                "value": "@pipeline().parameters.JobId",
                                "type": "Expression"
                            },
                            "type": "Int32"
                        }
                    }
                },
                "linkedServiceName": {
                    "referenceName": "AzureScheduleDataFactoryLink",
                    "type": "LinkedServiceReference"
                }
            }
        ],
         "parameters": {
            "JobId": {
                "type": "int",
                "defaultValue": 30691
            }
        },
        "annotations": [],
        "lastPublishTime": "2024-04-22T13:48:59Z"
    }
}

Clinton Morgan 0

Json for the creation of the trigger
URL for the PUT request
"/management.azure.com/subscriptions/ID/resourceGroups/ID/providers/Microsoft.DataFactory/factories/DataFactoryName/triggers/CM_Trigger1?api-version=2018-06-01"
Body



{
    "name": "CM_Trigger1",
    "properties": {
        "type": "ScheduleTrigger",
        "typeProperties": {
            "recurrence": {
                "frequency": "Minute",
                "interval": 10,
                "startTime": "2024-04-23T11:00:00.8441801Z",
                "timeZone": "GMT Standard Time"
            }
        }, 
        
        "pipelines": [
            {
                "pipelineReference": {
                "referenceName": "CM_ReportScheduler1"
                },
                "parameters": {}
            }
        ],
    "annotations": [],
    "description": "Publishing trigger changes"

    }
}

BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2024-04-26T20:04:26.6333333+00:00

Hello Clinton ,

I checked with my internal team and here is the suggestion.

first try to create trigger using API without providing any pipeline reference (https://learn.microsoft.com/en-us/rest/api/datafactory/triggers/create-or-update?view=rest-datafactory-2018-06-01&tabs=HTTP)
and after creation is successful, then start trigger using API if it is started then update the same trigger by adding pipeline reference then start the trigger using API again(https://learn.microsoft.com/en-us/rest/api/datafactory/triggers/start?view=rest-datafactory-2018-06-01&tabs=HTTP)

Please try the above and let me know.
Clinton Morgan 0 Reputation points

2024-04-29T09:45:54.1133333+00:00
Hi BhargavaGunnam-MSFT, thanks for getting back to me.
I followed your suggestions and still no luck.
Created the Trigger and tried to start it, but came back with the following error.

{ "error": { "code": "BadRequest", "message": "Missing or invalid pipeline references for trigger CM_Trigger2", "target": "CM_Trigger2", "details": null } }

I can see the trigger on Azure and its in a stopped status,
If I manually start then publish, manually stop and publish it still has a problem, If I then run the API with a reference to the pipeline it still wont work, But If I then manually start publish then manually stop then publish with the trigger linked to the pipeline, then run the API call to start the trigger it works.

It seems it wants it to be published manually before the API start will have an effect.

Is it maybe some permissions that I don't know about, that will let the API Start have the ability to publish the trigger and start it in the same call?

Thank once again for the help Bhargava.
BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2024-04-29T19:52:49+00:00

Hello Clinton,

Thanks for the additional details. This seems to be a bug or a permissions issue.

I suggest filing a support request so that a support engineer can further troubleshoot the issue during a screen-sharing session. If you don't have a support plan, please let me know. I can provide a one-time free support request to work on this case.

I look forward to hearing from you.