Invoke-AzDataFactoryV2Pipeline unable to run

Question

Invoke-AzDataFactoryV2Pipeline unable to run

Dong, James 57

In PowerShell Azure Module, if try to run Invoke-AzDataFactoryV2Pipeline cmdlet, no matter what is input into, existing or not existing, it always complain "Object reference not set to an instance of an object". Can you please help how to make this cmdlet work.

Below is an example of error:

PS C:\apps\dev\comm> Invoke-AzDataFactoryV2Pipeline

cmdlet Invoke-AzDataFactoryV2Pipeline at command pipeline position 1
Supply values for the following parameters:
(Type !? for Help.)
ResourceGroupName: test
DataFactoryName: test
PipelineName: test
Invoke-AzDataFactoryV2Pipeline : Object reference not set to an instance of an object.
At line:1 char:1

Invoke-AzDataFactoryV2Pipeline
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CategoryInfo : CloseError: (:) [Invoke-AzDataFactoryV2Pipeline], NullReferenceException
FullyQualifiedErrorId : Microsoft.Azure.Commands.DataFactoryV2.InvokeAzureDataFactoryPipelineCommand

Dong, James 57 Reputation points

2022-12-07T14:24:51.507+00:00

Can someone help what is the problem for this azure module for PowerShell for data factory? Specifically for this cmdlet: Invoke-AzDataFactoryV2Pipeline
MartinJaffer-MSFT 26,236 Reputation points

2022-12-07T20:30:22.747+00:00
Hello @Anonymous and welcome to Microsoft Q&A

In the past, I saw a similar error message caused by a bug in the asset definition. I don't think this is the case here. You asserted you tried with different pipelines, both ones existing and not, and got the same message. Trying to run a missing pipeline should have a different error message.

If the pipeline does not exist, the error message should looke like

Invoke-AzDataFactoryV2Pipeline: HTTP Status Code: BadRequest Error Code: BadRequest Error Message: Entity abcd not found

This discrepancy makes me think you are using an outdated or broken version of the Az module.

To verify this is an issue just with the Powershell, and not with the pipelines, can you please try going to the Studio, and triggering a run of the pipeline from Live mode (not Git / Devops mode).

Thank you.
Dong, James 57 Reputation points

2022-12-08T14:20:03.113+00:00

Tried get latest 1.16.11 version of az.datafactory module, and also tried in both PowerShell 5.1 and 7.3 version, all get same error. It seems nothing related to what parameters input, and it really hasn't reach to ADF instance yet. It always complain the same error from the module itself. We are running it on windows 2022.

We did try in cloud PowerShell (same 7.3 version with 1.16.11 az.datafactory) from Azure Portal which running on Ubuntu, it does return the real error which is expected.

cmdlet Invoke-AzDataFactoryV2Pipeline at command pipeline position 1
Supply values for the following parameters:
(Type !? for Help.)
ResourceGroupName: test
DataFactoryName: test
PipelineName: test
Invoke-AzDataFactoryV2Pipeline: HTTP Status Code: Forbidden
Error Code: AuthorizationFailed
Error Message: The client '******@cantire.com' with object id '3aebe5b8-e968-4c38-a110-0f8e61f99f5c' does not have authorization to perform action 'Microsoft.DataFactory/factories/pipelines/createRun/action' over scope '/subscriptions/e321efb2-bff7-4ec0-a4a7-c5f95abef641/resourceGroups/test/providers/Microsoft.DataFactory/factories/test/pipelines/test' or the scope is invalid. If access was recently granted, please refresh your credentials.
Request Id: d27830ca-f366-4509-8535-2daded7e5098
Timestamp (Utc):12/07/2022 19:16:55

Could this be a bug in PowerShell azure module for az.datafactory on windows?
MartinJaffer-MSFT 26,236 Reputation points

2022-12-08T16:47:36.71+00:00

@Anonymous thank you for the additional information. Having the cloud shell in the portal return different error makes this very interesting.

Now I am thinking one error is covering up another error. The lack of authorization is apermissions thing. Let us get past the permissions by going to the containing resource group, and assigning you owner permissions on the resource group. As the Factory is part of the resource group, it will inherit permissions. I am instructing to apply on the resource group first, rather than the Factory directly, because some edge cases require more than just the Factory. By applying owner to both the resource group and the Factory we cover all cases.

Once RBAC permissions are assigned, wait 15 minutes, then try the cloud shell again. Then try the original test.
If both tests work, then it means there is a bug in PowerShell giving the wrong error message.

I'll consult with my colleague who does more with the powershell and api stuff.
Dong, James 57 Reputation points

2022-12-08T17:01:42.207+00:00

Unfortunately I cannot let cloud shell access to the resource in our vnet. I'm expected error from cloud shell which is correct, it already proved the module works fine, just no permission or no path to reach to the ADF resource.

Which I don't expect is the error message from PowerShell module running on our windows 2022 VM. It even hasn't done anything yet then just complain back "Object reference not set to an instance of an object". For me, it seems even didn't pass the input parameter validation. If it passed, and try to connect, it should come back the same error messages like the one from cloud shell.

Can you please help that consult you mentioned?

Thanks,
James

1 answer

Your answer

Dong, James 57 Reputation points

2022-12-07T14:24:51.507+00:00

Can someone help what is the problem for this azure module for PowerShell for data factory? Specifically for this cmdlet: Invoke-AzDataFactoryV2Pipeline
MartinJaffer-MSFT 26,236 Reputation points

2022-12-07T20:30:22.747+00:00

Hello @Anonymous and welcome to Microsoft Q&A

In the past, I saw a similar error message caused by a bug in the asset definition. I don't think this is the case here. You asserted you tried with different pipelines, both ones existing and not, and got the same message. Trying to run a missing pipeline should have a different error message.

If the pipeline does not exist, the error message should looke like

Invoke-AzDataFactoryV2Pipeline: HTTP Status Code: BadRequest Error Code: BadRequest Error Message: Entity abcd not found

This discrepancy makes me think you are using an outdated or broken version of the Az module.

To verify this is an issue just with the Powershell, and not with the pipelines, can you please try going to the Studio, and triggering a run of the pipeline from Live mode (not Git / Devops mode).

Thank you.
Dong, James 57 Reputation points

2022-12-08T14:20:03.113+00:00

Tried get latest 1.16.11 version of az.datafactory module, and also tried in both PowerShell 5.1 and 7.3 version, all get same error. It seems nothing related to what parameters input, and it really hasn't reach to ADF instance yet. It always complain the same error from the module itself. We are running it on windows 2022.

We did try in cloud PowerShell (same 7.3 version with 1.16.11 az.datafactory) from Azure Portal which running on Ubuntu, it does return the real error which is expected.

cmdlet Invoke-AzDataFactoryV2Pipeline at command pipeline position 1
Supply values for the following parameters:
(Type !? for Help.)
ResourceGroupName: test
DataFactoryName: test
PipelineName: test
Invoke-AzDataFactoryV2Pipeline: HTTP Status Code: Forbidden
Error Code: AuthorizationFailed
Error Message: The client '******@cantire.com' with object id '3aebe5b8-e968-4c38-a110-0f8e61f99f5c' does not have authorization to perform action 'Microsoft.DataFactory/factories/pipelines/createRun/action' over scope '/subscriptions/e321efb2-bff7-4ec0-a4a7-c5f95abef641/resourceGroups/test/providers/Microsoft.DataFactory/factories/test/pipelines/test' or the scope is invalid. If access was recently granted, please refresh your credentials.
Request Id: d27830ca-f366-4509-8535-2daded7e5098
Timestamp (Utc):12/07/2022 19:16:55

Could this be a bug in PowerShell azure module for az.datafactory on windows?
MartinJaffer-MSFT 26,236 Reputation points

2022-12-08T16:47:36.71+00:00

@Anonymous thank you for the additional information. Having the cloud shell in the portal return different error makes this very interesting.

Now I am thinking one error is covering up another error. The lack of authorization is apermissions thing. Let us get past the permissions by going to the containing resource group, and assigning you owner permissions on the resource group. As the Factory is part of the resource group, it will inherit permissions. I am instructing to apply on the resource group first, rather than the Factory directly, because some edge cases require more than just the Factory. By applying owner to both the resource group and the Factory we cover all cases.

Once RBAC permissions are assigned, wait 15 minutes, then try the cloud shell again. Then try the original test.
If both tests work, then it means there is a bug in PowerShell giving the wrong error message.

I'll consult with my colleague who does more with the powershell and api stuff.
Dong, James 57 Reputation points

2022-12-08T17:01:42.207+00:00

Unfortunately I cannot let cloud shell access to the resource in our vnet. I'm expected error from cloud shell which is correct, it already proved the module works fine, just no permission or no path to reach to the ADF resource.

Which I don't expect is the error message from PowerShell module running on our windows 2022 VM. It even hasn't done anything yet then just complain back "Object reference not set to an instance of an object". For me, it seems even didn't pass the input parameter validation. If it passed, and try to connect, it should come back the same error messages like the one from cloud shell.

Can you please help that consult you mentioned?

Thanks,
James

Answer 1

Rich Matheisen 47,901

Looking at the syntax for that cmdlet, it appears that there are one of two sets of parameters required. You can supply these three:

[-ResourceGroupName] <String>
[-DataFactoryName] <String>
[-PipelineName] <String>

Or this one:

[-InputObject] <PSPipeline>

Is it possible that one (or more) of the string values you've supplied is not named "test"?

Can you use the Get-AzDataFactoryV2Pipeline cmdlet to get the desired pipeline and pipe the object to the Invoke-AzDataFactoryV2Pipeline?

You should also check the version of the module. Perhaps there's a newer version available.

Dong, James 57 Reputation points

2022-12-08T14:40:06.667+00:00
The Get-AzDataFactoryV2Pipeline have the same error. Tried real pipeline name, and also tried below dummy test, get same error for latest version of module and for both PowerShell 5.1 and 7.3.

PS C:\Users\James.Dong> Get-AzDataFactoryV2Pipeline

cmdlet Get-AzDataFactoryV2Pipeline at command pipeline position 1
Supply values for the following parameters:
(Type !? for Help.)
ResourceGroupName: test
DataFactoryName: test
Get-AzDataFactoryV2Pipeline : Object reference not set to an instance of an object.
At line:1 char:1

Get-AzDataFactoryV2Pipeline

~~~~~~~~~~~~~~~~~~~~~~~~~~~

CategoryInfo : CloseError: (:) [Get-AzDataFactoryV2Pipeline], NullReferenceException

FullyQualifiedErrorId : Microsoft.Azure.Commands.DataFactoryV2.GetAzureDataFactoryPipelineCommand
Rich Matheisen 47,901 Reputation points

2022-12-08T19:31:32.73+00:00

The Invoke-AzDataFactoryV2Pipeline cmdlet supports the use of "Common Parameters". Does adding "-ErrorAction STOP" to the cmdlet produce a more meaningful error message?
Dong, James 57 Reputation points

2022-12-08T19:59:56.023+00:00

Added "-ErrorAction STOP" but still got same error, no meaningful error message.
MartinJaffer-MSFT 26,236 Reputation points

2022-12-14T18:04:48.157+00:00

@Anonymous

The Get-AzDataFactoryPipeline shouldn't make this error, I think. It is sounding like any/all of the commandletts are giving this error, which sugguests the problem is something deeper like unable to connect.
Dong, James 57 Reputation points

2022-12-14T18:14:06.047+00:00

The issue is resolved. The error message is misleading, the actual problem is require the reader permission on the resource group for the user or managed identity who will execute the pipeline. Thanks everyone for the prompt help.
MartinJaffer-MSFT 26,236 Reputation points

2022-12-16T20:32:37.927+00:00

Thank you for sharing the resolution!

Share via

Invoke-AzDataFactoryV2Pipeline unable to run

1 answer

Your answer