Automated publishing of Data Factory with DevOps

Question

Automated publishing of Data Factory with DevOps

Avin Mathew 21

I've been following the instructions in Automated publishing for continuous integration and delivery to build the ARM templates for Data Factory.

The Validate job seems to run successfully in Azure DevOps, but I do see some errors in the logs:

ERROR === LocalFileClientService: Unable to read file: /home/vsts/work/1/s/publish_config.json, error: {"stack":"Error: ENOENT: no such file or directory, open '/home/vsts/work/1/s/publish_config.json'","message":"ENOENT: no such file or directory, open '/home/vsts/work/1/s/publish_config.json'","errno":-2,"code":"ENOENT","syscall":"open","path":"/home/vsts/work/1/s/publish_config.json"} ERROR === PublishConfigService: _getLatestPublishConfig - Unable to process publish config file, error: {"stack":"Error: ENOENT: no such file or directory, open '/home/vsts/work/1/s/publish_config.json'","message":"ENOENT: no such file or directory, open '/home/vsts/work/1/s/publish_config.json'","errno":-2,"code":"ENOENT","syscall":"open","path":"/home/vsts/work/1/s/publish_config.json"}

The Validate and Generate ARM template job fails with:

ERROR === LocalFileClientService: Unable to read file: /home/vsts/work/1/s/arm-template-parameters-definition.json, error: {"stack":"Error: ENOENT: no such file or directory, open '/home/vsts/work/1/s/arm-template-parameters-definition.json'","message":"ENOENT: no such file or directory, open '/home/vsts/work/1/s/arm-template-parameters-definition.json'","errno":-2,"code":"ENOENT","syscall":"open","path":"/home/vsts/work/1/s/arm-template-parameters-definition.json"}

and many errors like:

ERROR === BaseAsyncAdapter: {"model":"Copy","name":"Load data","error":"toResource failed. Resason: {\"stack\":\"NullInjectorError: StaticInjectorError[e]: \\n NullInjectorError: No provider for e!\\n at e.get (/home/vsts/work/1/s/downloads/main.js:76:1120)\\n at /home/vsts/work/1/s/downloads/main.js:398:2458\\n at Eo (/home/vsts/work/1/s/downloads/main.js:398:2865)\\n at e.get (/home/vsts/work/1/s/downloads/main.js:398:2031)\\n at t.<anonymous> (/home/vsts/work/1/s/downloads/main.js:1058:923482)\\n at /home/vsts/work/1/s/downloads/main.js:16:2243\\n at Object.next (/home/vsts/work/1/s/downloads/main.js:16:2348)\\n at /home/vsts/work/1/s/downloads/main.js:16:1285\\n at new Promise (<anonymous>)\\n at Object.l (/home/vsts/work/1/s/downloads/main.js:16:1030)\",\"message\":\"StaticInjectorError[e]: \\n NullInjectorError: No provider for e!\",\"name\":\"NullInjectorError\",\"ngTempTokenPath\":null,\"ngTokenPath\":[null]}"}

I also tried to run the npm commands locally, but Validate fails with:

ERROR === CmdApiApp: Failed to run resource validation. Error: {"stack":"TypeError: Cannot read property 'concept' of undefined\n at Function.<anonymous> (C:\\src\\data-factory\\downloads\\main.js:1271:27513)\n at C:\\src\\data-factory\\downloads\\main.js:16:2243\n at Object.next (C:\\src\\data-factory\\downloads\\main.js:16:2348)\n at o (C:\\src\\data-factory\\downloads\\main.js:16:1087)","message":"Cannot read property 'concept' of undefined"}

Any ideas on what is going wrong?

Saurabh Sharma 23,846 Reputation points Microsoft Employee Moderator

2021-02-23T01:44:02.623+00:00

@Avin Mathew Thanks for using Microsoft Q&A !!

I believe this issue requires deeper investigation from a support engineer in your environment, so I suggest you to please open an Azure support ticket for the same. In case you have any limitations creating a support ticket please send an email to azcommunity[at]microsoft[dot]com with your subscription id with this Q&A thread link and subject as "Attn: Saurabh' and I will help providing a free support ticket for you.
Saurabh Sharma 23,846 Reputation points Microsoft Employee Moderator

2021-02-24T15:58:32.29+00:00

@Avin Mathew Please let me know if need any help to create a support ticket.
Avin Mathew 21 Reputation points

2021-02-24T21:48:55.983+00:00

@Saurabh Sharma I had created a support ticket via Azure Portal: #2102230020000435
Saurabh Sharma 23,846 Reputation points Microsoft Employee Moderator

2021-02-25T17:17:59.863+00:00

@Avin Mathew Thanks for the confirmation.
Nick Hopkins 1 Reputation point

2021-03-17T21:07:18.22+00:00

Any resolution so far? I'm getting the same ENOENT: no such file or directory, open '/home/vsts/work/1/s/publish_config.json' error in my validate and publish steps.
Avin Mathew 21 Reputation points

2021-03-17T21:12:19.51+00:00

Microsoft indicated that I had to manually create the publish_config.json file even though I was using the default adf_publish branch. See https://learn.microsoft.com/en-us/azure/data-factory/source-control#version-control.

They also fixed a bug in the NPM package to fix the other errors I was receiving, so I just had to update the version of the package.
extvindad 1 Reputation point

2021-06-28T12:06:11.447+00:00

Can I ask for resolution your 4th error? What is related to? Thank you

ERROR === CmdApiApp: Failed to run resource validation. Error: {"stack":"TypeError: Cannot read property 'concept' of undefined\n at Function.<anonymous> (/home/vsts/work/1/s/DataFactory/downloads/main.js:1271:37748)\n at /home/vsts/work/1/s/DataFactory/downloads/main.js:16:2243\n at Object.next (/home/vsts/work/1/s/DataFactory/downloads/main.js:16:2348)\n at /home/vsts/work/1/s/DataFactory/downloads/main.js:16:1285\n at new Promise (<anonymous>)\n at Object.u (/home/vsts/work/1/s/DataFactory/downloads/main.js:16:1030)\n at Function.t._validate (/home/vsts/work/1/s/DataFactory/downloads/main.js:1271:37309)\n at Function.<anonymous>
Schiphorst ter, D (Dick) 1 Reputation point

2021-06-28T12:47:53.747+00:00

Hey Mathew,

Unfortunately I have the same error. First off all I created the publish config file manually, but that didnt't worked for me. Do you have a working solution?

Please let me know

Kind regards,

Dickkieee

2 answers

Your answer

Saurabh Sharma 23,846 Reputation points Microsoft Employee Moderator

2021-02-23T01:44:02.623+00:00

@Avin Mathew Thanks for using Microsoft Q&A !!

I believe this issue requires deeper investigation from a support engineer in your environment, so I suggest you to please open an Azure support ticket for the same. In case you have any limitations creating a support ticket please send an email to azcommunity[at]microsoft[dot]com with your subscription id with this Q&A thread link and subject as "Attn: Saurabh' and I will help providing a free support ticket for you.
Saurabh Sharma 23,846 Reputation points Microsoft Employee Moderator

2021-02-24T15:58:32.29+00:00

@Avin Mathew Please let me know if need any help to create a support ticket.
Avin Mathew 21 Reputation points

2021-02-24T21:48:55.983+00:00

@Saurabh Sharma I had created a support ticket via Azure Portal: #2102230020000435
Saurabh Sharma 23,846 Reputation points Microsoft Employee Moderator

2021-02-25T17:17:59.863+00:00

@Avin Mathew Thanks for the confirmation.
Nick Hopkins 1 Reputation point

2021-03-17T21:07:18.22+00:00

Any resolution so far? I'm getting the same ENOENT: no such file or directory, open '/home/vsts/work/1/s/publish_config.json' error in my validate and publish steps.
Avin Mathew 21 Reputation points

2021-03-17T21:12:19.51+00:00

Microsoft indicated that I had to manually create the publish_config.json file even though I was using the default adf_publish branch. See https://learn.microsoft.com/en-us/azure/data-factory/source-control#version-control.

They also fixed a bug in the NPM package to fix the other errors I was receiving, so I just had to update the version of the package.
extvindad 1 Reputation point

2021-06-28T12:06:11.447+00:00

Can I ask for resolution your 4th error? What is related to? Thank you

ERROR === CmdApiApp: Failed to run resource validation. Error: {"stack":"TypeError: Cannot read property 'concept' of undefined\n at Function.<anonymous> (/home/vsts/work/1/s/DataFactory/downloads/main.js:1271:37748)\n at /home/vsts/work/1/s/DataFactory/downloads/main.js:16:2243\n at Object.next (/home/vsts/work/1/s/DataFactory/downloads/main.js:16:2348)\n at /home/vsts/work/1/s/DataFactory/downloads/main.js:16:1285\n at new Promise (<anonymous>)\n at Object.u (/home/vsts/work/1/s/DataFactory/downloads/main.js:16:1030)\n at Function.t._validate (/home/vsts/work/1/s/DataFactory/downloads/main.js:1271:37309)\n at Function.<anonymous>
Schiphorst ter, D (Dick) 1 Reputation point

2021-06-28T12:47:53.747+00:00

Hey Mathew,

Unfortunately I have the same error. First off all I created the publish config file manually, but that didnt't worked for me. Do you have a working solution?

Please let me know

Kind regards,

Dickkieee

Answer 1

Manrique Vargas 6

Hello guys, this got solved. The leading '/' before 'subscriptions' is missing. Changing the Resource ID to /subscriptions/xxxxxxx/resourceGroups/xxxxxxx/providers/Microsoft.DataFactory/factories/xxxxxxxx should work. I tried again with the updated resource ID and it worked.
@Saurabh Sharma @DadejVincent-0237

Agelin, Jakob 21 Reputation points

2022-12-16T13:03:57.99+00:00

THANK YOU

Answer 2

Joost van Rossum 41

To solve the missing arm-template-parameters-definition.json you need to Edit the Parameter configuration within ADF. Then the message will disappear. See this

The missing publish_config.json can be solved by creating it in the repos (it looks like it isn't used any more, but somehow it keeps complaining about it). See this

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Share via

Automated publishing of Data Factory with DevOps

2 answers

Your answer