I wan use Show billing report by pipeline in ADF, and enabled this via CI/CD

Question

I wan use Show billing report by pipeline in ADF, and enabled this via CI/CD

Benjamin Baele 25 Microsoft Employee

I want enabled the feature "Show billing report by pipeline " in azure Data Factory, but I don't know how can I enabled this via Terraform deployment or ARM template deployment. How Can I do that ?

phemanth 15,765 Reputation points Microsoft External Staff Moderator

2024-07-03T16:51:57.2866667+00:00
@Benjamin Baele To enable the “Show billing report by pipeline” feature in Azure Data Factory, you can follow these steps

Go to the Azure Data Factory portal.

Under the Manage tab, select Factory settings in the General section.

Select Show billing report (preview) by pipeline.

Publish the change.

This allows pipeline-level billing data to begin aggregating for reporting. This setting is not included in the exported ARM templates from your factory, meaning that Continuous Integration and Delivery (CI/CD) will not overwrite billing behaviors for the factory. This allows you to set different billing behaviors for development, test, and production environments, even when they share the same pipeline definitions.

However, it’s important to note that as of now, there seems to be an issue where the “Show Billing report” setting is automatically reset to the value “by factory” when deploying the ARM of Data Factory by incremental deployment mode4. This issue has been reported and is awaiting a resolution.

please refer:https://learn.microsoft.com/en-us/azure/data-factory/plan-manage-costs

https://techcommunity.microsoft.com/t5/azure-data-factory-blog/granular-billing-for-azure-data-factory/ba-p/3654600

Hope this helps. Do let us know if you any further queries.
phemanth 15,765 Reputation points Microsoft External Staff Moderator

2024-07-04T06:31:09.1933333+00:00

@Benjamin Baele We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Benjamin Baele 25 Reputation points Microsoft Employee

2024-07-04T09:04:51.5433333+00:00

Thank you for these informations, but your answer is not entirely satisfactory. I asked how to implement CI/CD deployment to propagate the activation of this feature other than by manual intervention in the Azure portal.

Is it possible by ARM template (not generated by incremental deployment) orbetter with Terraform ?

I understand that for now it still a bug waiting for a resolution.
phemanth 15,765 Reputation points Microsoft External Staff Moderator

2024-07-05T09:41:00.1666667+00:00
@Benjamin Baele

You're right, achieving fully automated CI/CD deployment for "Show billing report by pipeline" in ADF with ARM templates or Terraform currently has limitations. Here's a breakdown of the situation and potential solutions:

Current Limitations:

ARM templates (even non-incrementally deployed) cannot directly configure the "Show billing report by pipeline" setting.

Terraform, while offering Infrastructure as Code (IaC) benefits, also lacks direct support for this specific setting.

There's a known bug where incremental ARM deployments might reset the setting.

Potential Solutions (Workarounds):

Separate ADF Instances: While not ideal, consider creating separate ADF instances for each environment (dev, test, prod). Manually configure the billing report setting in each instance for granular control. This approach avoids relying on CI/CD for this specific setting but requires managing multiple ADFs.

CI/CD with Scripting: Although ARM/Terraform can't directly manage the setting, you can integrate scripting into your CI/CD pipeline. Here's the idea:

Deploy your ADF pipelines using ARM/Terraform.

After deployment (potentially in a post-deployment script), use Azure CLI commands (or other tools) to access the ADF instance and programmatically enable the "Show billing report by pipeline" setting.

Downsides of Scripting Approach:

Requires additional scripting and maintenance effort.

Scripting tools might introduce platform-specific dependencies.

Future Outlook:

Microsoft is aware of the ARM deployment bug and is working on a resolution. Hopefully, future updates will allow ARM templates to manage this setting directly. manage this setting directly.

Benjamin Baele 25 Microsoft Employee

Infortunatly Azure CLI doesn't not support this feature also: https://learn.microsoft.com/en-us/cli/azure/datafactory?view=azure-cli-latest#az-datafactory-update

I find a solution with REST API call, executed by Powershell. I share my script to help other users:

param(
    [string]$subscriptionId,
    [string]$resourceGroupName,
    [string]$dataFactoryName,
    [bool]$pipelineBillingEnabled = $true
)

$uri = "https://management.azure.com/subscriptions/" + $subscriptionId +  "/resourceGroups/" + $resourceGroupName + "/providers/Microsoft.DataFactory/factories/" + $dataFactoryName + "?api-version=2018-06-01"

$body = @{
    properties = @{
        globalConfigurations = @{
            PipelineBillingEnabled = $pipelineBillingEnabled
        }
    }
} | ConvertTo-Json

$token = (Get-AzAccessToken -ResourceUrl "https://management.azure.com").Token

$headers = @{
    "Authorization" = "Bearer $token"
    "Content-Type" = "application/json"
}

$response = Invoke-WebRequest -Uri $uri -Method Patch -Headers $headers -Body $body 

if ($response.StatusCode -eq 200) {
    Write-Host "Data Factory resource updated successfully."
} else {
    Write-Host "Failed to update Data Factory resource. Status code: $($response.StatusCode)"
}

phemanth 15,765 Reputation points Microsoft External Staff Moderator

2024-07-06T04:51:16.6333333+00:00

@Benjamin Baele Glad to know your issue has been resolved. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Accepted answer

0 additional answers

Your answer

phemanth 15,765 Reputation points Microsoft External Staff Moderator

2024-07-03T16:51:57.2866667+00:00

@Benjamin Baele To enable the “Show billing report by pipeline” feature in Azure Data Factory, you can follow these steps

Go to the Azure Data Factory portal.

Under the Manage tab, select Factory settings in the General section.

Select Show billing report (preview) by pipeline.

Publish the change.

This allows pipeline-level billing data to begin aggregating for reporting. This setting is not included in the exported ARM templates from your factory, meaning that Continuous Integration and Delivery (CI/CD) will not overwrite billing behaviors for the factory. This allows you to set different billing behaviors for development, test, and production environments, even when they share the same pipeline definitions.

However, it’s important to note that as of now, there seems to be an issue where the “Show Billing report” setting is automatically reset to the value “by factory” when deploying the ARM of Data Factory by incremental deployment mode4. This issue has been reported and is awaiting a resolution.

please refer:https://learn.microsoft.com/en-us/azure/data-factory/plan-manage-costs

https://techcommunity.microsoft.com/t5/azure-data-factory-blog/granular-billing-for-azure-data-factory/ba-p/3654600

Hope this helps. Do let us know if you any further queries.
phemanth 15,765 Reputation points Microsoft External Staff Moderator

2024-07-04T06:31:09.1933333+00:00

@Benjamin Baele We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Benjamin Baele 25 Reputation points Microsoft Employee

2024-07-04T09:04:51.5433333+00:00

Thank you for these informations, but your answer is not entirely satisfactory. I asked how to implement CI/CD deployment to propagate the activation of this feature other than by manual intervention in the Azure portal.

Is it possible by ARM template (not generated by incremental deployment) orbetter with Terraform ?

I understand that for now it still a bug waiting for a resolution.
phemanth 15,765 Reputation points Microsoft External Staff Moderator

2024-07-05T09:41:00.1666667+00:00

@Benjamin Baele

You're right, achieving fully automated CI/CD deployment for "Show billing report by pipeline" in ADF with ARM templates or Terraform currently has limitations. Here's a breakdown of the situation and potential solutions:

Current Limitations:

ARM templates (even non-incrementally deployed) cannot directly configure the "Show billing report by pipeline" setting.

Terraform, while offering Infrastructure as Code (IaC) benefits, also lacks direct support for this specific setting.

There's a known bug where incremental ARM deployments might reset the setting.

Potential Solutions (Workarounds):

Separate ADF Instances: While not ideal, consider creating separate ADF instances for each environment (dev, test, prod). Manually configure the billing report setting in each instance for granular control. This approach avoids relying on CI/CD for this specific setting but requires managing multiple ADFs.

CI/CD with Scripting: Although ARM/Terraform can't directly manage the setting, you can integrate scripting into your CI/CD pipeline. Here's the idea:

Deploy your ADF pipelines using ARM/Terraform.

After deployment (potentially in a post-deployment script), use Azure CLI commands (or other tools) to access the ADF instance and programmatically enable the "Show billing report by pipeline" setting.

Downsides of Scripting Approach:

Requires additional scripting and maintenance effort.

Scripting tools might introduce platform-specific dependencies.

Future Outlook:

Microsoft is aware of the ARM deployment bug and is working on a resolution. Hopefully, future updates will allow ARM templates to manage this setting directly. manage this setting directly.
Benjamin Baele 25 Reputation points Microsoft Employee

2024-07-05T12:20:37.4333333+00:00

Infortunatly Azure CLI doesn't not support this feature also: https://learn.microsoft.com/en-us/cli/azure/datafactory?view=azure-cli-latest#az-datafactory-update

I find a solution with REST API call, executed by Powershell. I share my script to help other users:

param( [string]$subscriptionId, [string]$resourceGroupName, [string]$dataFactoryName, [bool]$pipelineBillingEnabled = $true ) $uri = "https://management.azure.com/subscriptions/" + $subscriptionId + "/resourceGroups/" + $resourceGroupName + "/providers/Microsoft.DataFactory/factories/" + $dataFactoryName + "?api-version=2018-06-01" $body = @{ properties = @{ globalConfigurations = @{ PipelineBillingEnabled = $pipelineBillingEnabled } } } | ConvertTo-Json $token = (Get-AzAccessToken -ResourceUrl "https://management.azure.com").Token $headers = @{ "Authorization" = "Bearer $token" "Content-Type" = "application/json" } $response = Invoke-WebRequest -Uri $uri -Method Patch -Headers $headers -Body $body if ($response.StatusCode -eq 200) { Write-Host "Data Factory resource updated successfully." } else { Write-Host "Failed to update Data Factory resource. Status code: $($response.StatusCode)" }
phemanth 15,765 Reputation points Microsoft External Staff Moderator

2024-07-06T04:51:16.6333333+00:00

@Benjamin Baele Glad to know your issue has been resolved. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Answer 1

@Benjamin Baele

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Ask: Trying to create a schema for an external table on the serverless SQL pool in Azure Synapse Analytics. The error message they received indicates that the specified schema name either does not exist or they do not have permission to use it.

Solution: Infortunatly Azure CLI doesn't not support this feature also: https://learn.microsoft.com/en-us/cli/azure/datafactory?view=azure-cli-latest#az-datafactory-update

I find a solution with REST API call, executed by Powershell. I share my script to help other users:

param(
    [string]$subscriptionId,
    [string]$resourceGroupName,
    [string]$dataFactoryName,
    [bool]$pipelineBillingEnabled = $true
)
$uri = "https://management.azure.com/subscriptions/" + $subscriptionId +  "/resourceGroups/" + $resourceGroupName + "/providers/Microsoft.DataFactory/factories/" + $dataFactoryName + "?api-version=2018-06-01"
$body = @{
    properties = @{
        globalConfigurations = @{
            PipelineBillingEnabled = $pipelineBillingEnabled
        }
    }
} | ConvertTo-Json
$token = (Get-AzAccessToken -ResourceUrl "https://management.azure.com").Token
$headers = @{
    "Authorization" = "Bearer $token"
    "Content-Type" = "application/json"
}
$response = Invoke-WebRequest -Uri $uri -Method Patch -Headers $headers -Body $body 
if ($response.StatusCode -eq 200) {
    Write-Host "Data Factory resource updated successfully."
} else {
    Write-Host "Failed to update Data Factory resource. Status code: $($response.StatusCode)"
}

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Was this answer helpful?Yes

PK 5 Reputation points

2024-07-17T13:59:15.7466667+00:00

Thanks for the solution! Note: PipelineBillingEnabled is a string, not a bool. You can set a bool but the UI doesn't like it

Share via

I wan use Show billing report by pipeline in ADF, and enabled this via CI/CD

0 additional answers

Your answer