Azure Data Factory GitHub Actions getting the error message "InvalidContentLink"

Question

I'm stuck on the following error message:

Error: ERROR: {"code": "InvalidContentLink", "message": "Unable to download deployment content from '/ArmTemplate_0.jsonsig=Sanitized Azure Storage Account Shared Access Signature'. The tracking Id is 'f6b674ad-7dbc-4434-96ea-6e56cb34daa6'. Please see https://aka.ms/arm-deploy-resources for usage details."}

Originally we set up Azure Data Factory (ADF) using a GitHub repository. We then set up GitHub Actions to deploy our code base from our DEV environment to our PRD environment. We were able to get GitHub actions to work fine. But then the # of ADF Pipelines grew to over 200 and we're now getting this error message when running the GitHub deployment from DEV to PRD:

ERROR: {"code": "RequestContentTooLarge", "message": "The request content size exceeds the maximum size of 4 MB."}

I've read the documentation offered by Microsoft at these links:

Linked Resource Manager templates with CI/CD

https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-linked-templates

Deploying Linked ARM Templates with VSTS

https://learn.microsoft.com/en-us/archive/blogs/najib/deploying-linked-arm-templates-with-vsts

But it's not clear to me, how do specifically modify the YAML file presented in this technical document from Microsoft. https://techcommunity.microsoft.com/t5/fasttrack-for-azure/azure-data-factory-ci-cd-with-github-actions/bc-p/4232307#M1011

The above link worked originally just fine until the ADF grew to over 200 pipelines where the ARM template exceeded 4 MB. I know there's a 4 MB limit for ARM templates. And the documentation says to use linked templates.

I see in the "adf_publish" branch a subfolder called "linkedTemplates."

I did edit the YMAL file by replacing "ARMTemplateForFactory.json" with "linkedTemplates/ArmTemplate_master.json".

I then replaced "ARMTemplateParametersForFactory.json" with "linkedTemplates/ArmTemplateParameters_master.json".

I then created the two GitHub Action Secrets called "CONTAINER_URI" and "CONTAINER_SAS_TOKEN".

I then added this to the "additionalParameters" as follows

containerUri=${{ secrets.CONTAINER_URI }}

containerSasToken=${{ secrets.CONTAINER_SAS_TOKEN }}'

Also, for the MANAGED IDENTITY called "ADF-UAMI", I added the following role assignments:

Contributor

Storage Blob Data Contributor

Storage Account Contributor

Storage Blob Data Owner

Storage Blob Data Reader

Also, I generated the SAS Token in the storage account as prescribed in the instructions.

For reference, below is the YAML code I used, which gives me the error:

ERROR: {"code": "InvalidContentLink", "message": "Unable to download deployment content from '/ArmTemplate_0.jsonsig=Sanitized Azure Storage Account Shared Access Signature'. The tracking Id is 'f6b674ad-7dbc-4434-96ea-6e56cb34daa6'. Please see https://aka.ms/arm-deploy-resources for usage details."

Can someone let me know why I'm getting this error message?

Any help is greatly appreciated.

on:

workflow_dispatch:

branches:

- main

permissions:

id-token: write

contents: read

jobs:

build:

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v3

# Installs Node and the npm packages saved in your package.json file in the build

- name: Setup Node.js environment

uses: actions/setup-node@v3.4.1

with:

node-version: 18.x

- name: install ADF Utilities package

run: npm install

working-directory: ${{github.workspace}}/ADFroot/build # (1) provide the folder location of the package.json file

# Validates all of the Data Factory resources in the repository. You'll get the same validation errors as when "Validate All" is selected.

- name: Validate

run: npm run build validate ${{github.workspace}}/ADFroot/ /subscriptions/8d4d0831-455f-477c-a601-a497052fa6d0/resourceGroups/const-rg-edw-01/providers/Microsoft.DataFactory/factories/const-edw-dev-01 # (2) The validate command needs the root folder location of your repository where all the objects are stored. And the 2nd parameter is the resourceID of the ADF instance

working-directory: ${{github.workspace}}/ADFroot/build

- name: Validate and Generate ARM template

run: npm run build export ${{github.workspace}}/ADFroot/ /subscriptions/8d4d0831-455f-477c-a601-a497052fa6d0/resourceGroups/const-rg-edw-01/providers/Microsoft.DataFactory/factories/const-edw-dev-01 "ExportedArmTemplate" # (3) The build command, as validate, needs the root folder location of your repository where all the objects are stored. And the 2nd parameter is the resourceID of the ADF instance. The 3rd parameter is the exported ARM template artifact name

working-directory: ${{github.workspace}}/ADFroot/build

# In order to leverage the artifact in another job, we need to upload it with the upload action

- name: upload artifact

uses: actions/upload-artifact@v3

with:

name: ExportedArmTemplate # (4) use the same artifact name you used in the previous export step

path: ${{github.workspace}}/ADFroot/build/ExportedArmTemplate

release:

needs: build

runs-on: ubuntu-latest

steps:

# we 1st download the previously uploaded artifact so we can leverage it later in the release job

- name: Download a Build Artifact

uses: actions/download-artifact@v3.0.2

with:

name: ExportedArmTemplate # (5) Artifact name

- name: Login via Az module

uses: azure/login@v2

with:

client-id: ${{ secrets.AZURE_CLIENT_ID }}

tenant-id: ${{ secrets.AZURE_TENANT_ID }}

subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

enable-AzPSSession: true

- name: data-factory-deploy

uses: Azure/data-factory-deploy-action@v1.2.0

with:

resourceGroupName: const-rg-edw-01

dataFactoryName: const-edw-uat-01

armTemplateFile: linkedTemplates/ArmTemplate_master.json

armTemplateParametersFile: linkedTemplates/ArmTemplateParameters_master.json

additionalParameters:

'factoryName=const-edw-uat-01 containerUri=${{ secrets.CONTAINER_URI }} containerSasToken=${{ secrets.CONTAINER_SAS_TOKEN }}'

# skipAzModuleInstallation: # Parameters which skip the Az module installation. Optional, default is false.​

Answer 1

Below are answers to your questions:

1. I created the storage account and did not add any firewall rules. The storage account was created basically using all the defaults.
2. Anonymous access is disabled. But I do believe public access is enabled since we're not using any private endpoints.
3. Yes, that is my understanding that ARM needs to use the public internet. Note, I tried both "Microsoft network routing" and "Internet routing" which neither worked. I also checked the boxes for "Microsoft network routing" and "Internet routing" under "Publish route-specific endpoints" and that didn't work either. See the screenshot below.

I've tried many different scenarios and still no luck. I continue to get the same error message which is.

Inner Errors:

{"code": "InvalidContentLink", "message": "Unable to download deployment content from '/ArmTemplate_0.jsonsig=Sanitized Azure Storage Account Shared Access Signature'.

Answer 2

Below are my answers to your quesitons:

1. I just created the storage account using all the defaults. There are no specific IP addresses blocked. The default firewall is enabled as shown in the screenshot below.
2. Anonymous access to the storage account is disabled. But I do believe Public access is enabled. And we're not using private end-points.
3. I tried all the different options highlighted in the screenshot below. I tried both "Microsoft network routing" and "Internet routing". Also, I checked the boxes for "Microsoft network routing" and "Internet routing". This did not make a difference.

I continue to get the same error over and over which is:

Inner Errors:

{"code": "InvalidContentLink", "message": "Unable to download deployment content from '/ArmTemplate_0.jsonsig=Sanitized Azure Storage Account Shared Access Signature'. The tracking Id is 'd2ea1d06-820b-4c69-86de-68e58728b10c'. Please see https://aka.ms/arm-deploy-resources for usage details."}

Here's the screenshot: