Tutorial: Implement CI/CD with GitOps (Flux v2)

In this tutorial, you'll set up a CI/CD solution using GitOps with Flux v2 and Azure Arc-enabled Kubernetes or Azure Kubernetes Service (AKS) clusters. Using the sample Azure Vote app, you'll:

  • Create an Azure Arc-enabled Kubernetes or AKS cluster.
  • Connect your application and GitOps repositories to Azure Repos or GitHub.
  • Implement CI/CD flow with either Azure Pipelines or GitHub.
  • Connect your Azure Container Registry to Azure DevOps and Kubernetes.
  • Create environment variable groups or secrets.
  • Deploy the dev and stage environments.
  • Test the application environments.

Note

Eventually Azure will stop supporting GitOps with Flux v1, so begin using Flux v2 as soon as possible.

If you don't have an Azure subscription, create a free account before you begin.

Azure Cloud Shell

Azure hosts Azure Cloud Shell, an interactive shell environment that you can use through your browser. You can use either Bash or PowerShell with Cloud Shell to work with Azure services. You can use the Cloud Shell preinstalled commands to run the code in this article, without having to install anything on your local environment.

To start Azure Cloud Shell:

Option Example/Link
Select Try It in the upper-right corner of a code or command block. Selecting Try It doesn't automatically copy the code or command to Cloud Shell. Screenshot that shows an example of Try It for Azure Cloud Shell.
Go to https://shell.azure.com, or select the Launch Cloud Shell button to open Cloud Shell in your browser. Screenshot that shows how to launch Cloud Shell in a new window.
Select the Cloud Shell button on the menu bar at the upper right in the Azure portal. Screenshot that shows the Cloud Shell button in the Azure portal

To use Azure Cloud Shell:

  1. Start Cloud Shell.

  2. Select the Copy button on a code block (or command block) to copy the code or command.

  3. Paste the code or command into the Cloud Shell session by selecting Ctrl+Shift+V on Windows and Linux, or by selecting Cmd+Shift+V on macOS.

  4. Select Enter to run the code or command.

Prerequisites

  • Complete the previous tutorial to learn how to deploy GitOps for your CI/CD environment.

  • Understand the benefits and architecture of this feature.

  • Verify you have:

  • Install the latest versions of these Azure Arc-enabled Kubernetes and Kubernetes Configuration CLI extensions:

    az extension add --name connectedk8s
    az extension add --name k8s-configuration
    
    • To update these extensions to the latest version, run the following commands:

      az extension update --name connectedk8s
      az extension update --name k8s-configuration
      

Connect Azure Container Registry to Kubernetes

Enable your Kubernetes cluster to pull images from your Azure Container Registry. If it's private, authentication will be required.

Connect Azure Container Registry to existing AKS clusters

Integrate an existing Azure Container Registry with existing AKS clusters using the following command:

az aks update -n arc-cicd-cluster -g myResourceGroup --attach-acr arc-demo-acr

Create an image pull secret

To connect non-AKS and local clusters to your Azure Container Registry, create an image pull secret. Kubernetes uses image pull secrets to store information needed to authenticate your registry.

Create an image pull secret with the following kubectl command. Repeat for both the dev and stage namespaces.

kubectl create secret docker-registry <secret-name> \
    --namespace <namespace> \
    --docker-server=<container-registry-name>.azurecr.io \
    --docker-username=<service-principal-ID> \
    --docker-password=<service-principal-password>

To avoid having to set an imagePullSecret for every Pod, consider adding the imagePullSecret to the Service account in the dev and stage namespaces. See the Kubernetes tutorial for more information.

Depending on the CI/CD orchestrator you prefer, you can proceed with instructions either for Azure DevOps or for GitHub.

Implement CI/CD with Azure DevOps

This tutorial assumes familiarity with Azure DevOps, Azure Repos and Pipelines, and Azure CLI.

Make sure to complete the following:

Import application and GitOps repositories into Azure Repos

Import an application repository and a GitOps repository into Azure Repos. For this tutorial, use the following example repositories:

Learn more about importing Git repositories.

Note

Importing and using two separate repositories for application and GitOps repositories can improve security and simplicity. The application and GitOps repositories' permissions and visibility can be tuned individually. For example, the cluster administrator may not find the changes in application code relevant to the desired state of the cluster. Conversely, an application developer doesn't need to know the specific parameters for each environment - a set of test values that provide coverage for the parameters may be sufficient.

Connect the GitOps repository

To continuously deploy your app, connect the application repository to your cluster using GitOps. Your arc-cicd-demo-gitops GitOps repository contains the basic resources to get your app up and running on your arc-cicd-cluster cluster.

The initial GitOps repository contains only a manifest that creates the dev and stage namespaces corresponding to the deployment environments.

The GitOps connection that you create will automatically:

  • Sync the manifests in the manifest directory.
  • Update the cluster state.

The CI/CD workflow will populate the manifest directory with extra manifests to deploy the app.

  1. Create a new GitOps connection to your newly imported arc-cicd-demo-gitops repository in Azure Repos.

    az k8s-configuration flux create \
       --name cluster-config \
       --cluster-name arc-cicd-cluster \
       --namespace flux-system \
       --resource-group myResourceGroup \
       -u https://dev.azure.com/<Your organization>/<Your project>/_git/arc-cicd-demo-gitops \
       --https-user <Azure Repos username> \
       --https-key <Azure Repos PAT token> \
       --scope cluster \
       --cluster-type managedClusters \
       --branch master \
       --kustomization name=cluster-config prune=true path=arc-cicd-cluster/manifests
    
  2. Check the state of the deployment in Azure portal.

    • If successful, you'll see both dev and stage namespaces created in your cluster.
    • You can also check on Azure Portal page of your K8s cluster on GitOps tab a configuration cluster-config is created.

Import the CI/CD pipelines

Now that you've synced a GitOps connection, you'll need to import the CI/CD pipelines that create the manifests.

The application repository contains a .pipeline folder with the pipelines you'll use for PRs, CI, and CD. Import and rename the three pipelines provided in the sample repository:

Pipeline file name Description
.pipelines/az-vote-pr-pipeline.yaml The application PR pipeline, named arc-cicd-demo-src PR
.pipelines/az-vote-ci-pipeline.yaml The application CI pipeline, named arc-cicd-demo-src CI
.pipelines/az-vote-cd-pipeline.yaml The application CD pipeline, named arc-cicd-demo-src CD

Connect Azure Container Registry to Azure DevOps

During the CI process, you'll deploy your application containers to a registry. Start by creating an Azure service connection:

  1. In Azure DevOps, open the Service connections page from the project settings page. In TFS, open the Services page from the settings icon in the top menu bar.
  2. Choose + New service connection and select the type of service connection you need.
  3. Fill in the parameters for the service connection. For this tutorial:
    • Name the service connection arc-demo-acr.
    • Select myResourceGroup as the resource group.
  4. Select the Grant access permission to all pipelines.
    • This option authorizes YAML pipeline files for service connections.
  5. Choose OK to create the connection.

Configure PR Service Connection

CD pipeline manipulates PRs in the GitOps repository. It needs a Service Connection for that:

  1. In Azure DevOps, open the Service connections page from the project settings page. In TFS, open the Services page from the settings icon in the top menu bar.
  2. Choose + New service connection and select Generic type.
  3. Fill in the parameters for the service connection. For this tutorial:
    • Server URL https://dev.azure.com/<Your organization>/<Your project>/_apis/git/repositories/arc-cicd-demo-gitops
    • Leave Username and Password blank
    • Name the service connection azdo-pr-connection.
  4. Select the Grant access permission to all pipelines.
    • This option authorizes YAML pipeline files for service connections.
  5. Choose OK to create the connection.

Install GitOps Connector

  1. Add GitOps Connector repository to Helm repositories:
   helm repo add gitops-connector https://azure.github.io/gitops-connector/
  1. Install the connector to the cluster:
   helm upgrade -i gitops-connector gitops-connector/gitops-connector \
      --namespace flux-system \
      --set gitRepositoryType=AZDO \
      --set ciCdOrchestratorType=AZDO \
      --set gitOpsOperatorType=FLUX \
      --set azdoGitOpsRepoName=arc-cicd-demo-gitops \
      --set azdoOrgUrl=https://dev.azure.com/<Your organization>/<Your project> \
      --set gitOpsAppURL=https://dev.azure.com/<Your organization>/<Your project>/_git/arc-cicd-demo-gitops \
      --set orchestratorPAT=<Azure Repos PAT token>

Note

Azure Repos PAT token should have Build: Read & execute and Code: Full permissions.

  1. Configure Flux to send notifications to GitOps connector:
cat <<EOF | kubectl apply -f -
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: gitops-connector
  namespace: flux-system
spec:
  eventSeverity: info
  eventSources:
  - kind: GitRepository
    name: cluster-config
  - kind: Kustomization
    name: cluster-config-cluster-config 
  providerRef:
    name: gitops-connector
---
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Provider
metadata:
  name: gitops-connector
  namespace: flux-system
spec:
  type: generic
  address: http://gitops-connector:8080/gitopsphase
EOF

For the details on installation, refer to the GitOps Connector repository.

Create environment variable groups

App repository variable group

Create a variable group named az-vote-app-dev. Set the following values:

Variable Value
AZ_ACR_NAME (your Azure Container Registry instance, for example. azurearctest.azurecr.io)
AZURE_SUBSCRIPTION (your Azure Service Connection, which should be arc-demo-acr from earlier in the tutorial)
AZURE_VOTE_IMAGE_REPO The full path to the Azure Vote App repository, for example azurearctest.azurecr.io/azvote
ENVIRONMENT_NAME Dev
MANIFESTS_BRANCH master
MANIFESTS_REPO arc-cicd-demo-gitops
ORGANIZATION_NAME Name of Azure DevOps organization
PROJECT_NAME Name of GitOps project in Azure DevOps
REPO_URL Full URL for GitOps repository
SRC_FOLDER azure-vote
TARGET_CLUSTER arc-cicd-cluster
TARGET_NAMESPACE dev
VOTE_APP_TITLE Voting Application
AKS_RESOURCE_GROUP AKS Resource group. Needed for automated testing.
AKS_NAME AKS Name. Needed for automated testing.

Stage environment variable group

  1. Clone the az-vote-app-dev variable group.
  2. Change the name to az-vote-app-stage.
  3. Ensure the following values for the corresponding variables:
Variable Value
ENVIRONMENT_NAME Stage
TARGET_NAMESPACE stage

You're now ready to deploy to the dev and stage environments.

Create environments

In Azure DevOps project create Dev and Stage environments. See Create and target an environment for more details.

Give more permissions to the build service

The CD pipeline uses the security token of the running build to authenticate to the GitOps repository. More permissions are needed for the pipeline to create a new branch, push changes, and create pull requests.

  1. Go to Project settings from the Azure DevOps project main page.
  2. Select Repos/Repositories.
  3. Select Security.
  4. For the <Project Name> Build Service (<Organization Name>) and for the Project Collection Build Service (<Organization Name>) (type in the search field, if it doesn't show up), allow Contribute, Contribute to pull requests, and Create branch.
  5. Go to Pipelines/Settings
  6. Switch off Protect access to repositories in YAML pipelines option

For more information, see:

Deploy the dev environment for the first time

With the CI and CD pipelines created, run the CI pipeline to deploy the app for the first time.

CI pipeline

During the initial CI pipeline run, you may get a resource authorization error in reading the service connection name.

  1. Verify the variable being accessed is AZURE_SUBSCRIPTION.
  2. Authorize the use.
  3. Rerun the pipeline.

The CI pipeline:

  • Ensures the application change passes all automated quality checks for deployment.
  • Does any extra validation that couldn't be completed in the PR pipeline.
    • Specific to GitOps, the pipeline also publishes the artifacts for the commit that will be deployed by the CD pipeline.
  • Verifies the Docker image has changed and the new image is pushed.

CD pipeline

During the initial CD pipeline run, you'll be asked to give the pipeline access to the GitOps repository. Select View when prompted that the pipeline needs permission to access a resource. Then, select Permit to grant permission to use the GitOps repository for the current and future runs of the pipeline.

The successful CI pipeline run triggers the CD pipeline to complete the deployment process. You'll deploy to each environment incrementally.

Tip

If the CD pipeline does not automatically trigger:

  1. Verify the name matches the branch trigger in .pipelines/az-vote-cd-pipeline.yaml
    • It should be arc-cicd-demo-src CI.
  2. Rerun the CI pipeline.

Once the template and manifest changes to the GitOps repository have been generated, the CD pipeline will create a commit, push it, and create a PR for approval.

  1. Find the PR created by the pipeline to the GitOps repository.

  2. Verify the changes to the GitOps repository. You should see:

    • High-level Helm template changes.
    • Low-level Kubernetes manifests that show the underlying changes to the desired state. Flux deploys these manifests.
  3. If everything looks good, approve and complete the PR.

  4. After a few minutes, Flux picks up the change and starts the deployment.

  5. Monitor the Git Commit status on the Commit history tab. Once it is succeeded the CD pipeline will go ahead and start automated testing

  6. Forward the port locally using kubectl and ensure the app works correctly using:

    kubectl port-forward -n dev svc/azure-vote-front 8080:80
    
  7. View the Azure Vote app in your browser at http://localhost:8080/.

  8. Vote for your favorites and get ready to make some changes to the app.

Set up environment approvals

Upon app deployment, you can not only make changes to the code or templates, but you can also unintentionally put the cluster into a bad state.

If the dev environment reveals a break after deployment, keep it from going to later environments using environment approvals.

  1. In your Azure DevOps project, go to the environment that needs to be protected.
  2. Navigate to Approvals and Checks for the resource.
  3. Select Create.
  4. Provide the approvers and an optional message.
  5. Select Create again to complete the addition of the manual approval check.

For more details, see the Define approval and checks tutorial.

Next time the CD pipeline runs, the pipeline will pause after the GitOps PR creation. Verify the change has been synced properly and passes basic functionality. Approve the check from the pipeline to let the change flow to the next environment.

Make an application change

With this baseline set of templates and manifests representing the state on the cluster, you'll make a small change to the app.

  1. In the arc-cicd-demo-src repository, edit azure-vote/src/azure-vote-front/config_file.cfg file.

  2. Since "Cats vs Dogs" isn't getting enough votes, change it to "Tabs vs Spaces" to drive up the vote count.

  3. Commit the change in a new branch, push it, and create a pull request.

    • This is the typical developer flow that will start the CI/CD lifecycle.

PR validation pipeline

The PR pipeline is the first line of defense against a faulty change. Usual application code quality checks include linting and static analysis. From a GitOps perspective, you also need to assure the same quality for the resulting infrastructure to be deployed.

The application's Dockerfile and Helm charts can use linting in a similar way to the application.

Errors found during linting range from:

  • Incorrectly formatted YAML files, to
  • Best practice suggestions, such as setting CPU and Memory limits for your application.

Note

To get the best coverage from Helm linting in a real application, you will need to substitute values that are reasonably similar to those used in a real environment.

Errors found during pipeline execution appear in the test results section of the run. From here, you can:

  • Track the useful statistics on the error types.
  • Find the first commit on which they were detected.
  • Stack trace style links to the code sections that caused the error.

Once the pipeline run has finished, you have assured the quality of the application code and the template that will deploy it. You can now approve and complete the PR. The CI will run again, regenerating the templates and manifests, before triggering the CD pipeline.

Tip

In a real environment, don't forget to set branch policies to ensure the PR passes your quality checks. For more information, see the Set branch policies article.

CD process approvals

A successful CI pipeline run triggers the CD pipeline to complete the deployment process. This time, the pipeline requires you to approve each deployment environment.

  1. Approve the deployment to the dev environment.
  2. Once the template and manifest changes to the GitOps repository have been generated, the CD pipeline will create a commit, push it, and create a PR for approval.
  3. Verify the changes to the GitOps repository. You should see:
    • High-level Helm template changes.
    • Low-level Kubernetes manifests that show the underlying changes to the desired state.
  4. If everything looks good, approve and complete the PR.
  5. Wait for the deployment to complete.
  6. As a basic smoke test, navigate to the application page and verify the voting app now displays Tabs vs Spaces.
    • Forward the port locally using kubectl and ensure the app works correctly using: kubectl port-forward -n dev svc/azure-vote-front 8080:80
    • View the Azure Vote app in your browser at http://localhost:8080/ and verify the voting choices have changed to Tabs vs Spaces.
  7. Repeat steps 1-7 for the stage environment.

Your deployment is now complete. This ends the CI/CD workflow. Refer to the Azure DevOps GitOps Flow diagram in the application repository that explains in details the steps and techniques implemented in the CI/CD pipelines used in this tutorial.

Implement CI/CD with GitHub

This tutorial assumes familiarity with GitHub, GitHub Actions.

Fork application and GitOps repositories

Fork an application repository and a GitOps repository. For this tutorial, use the following example repositories:

Connect the GitOps repository

To continuously deploy your app, connect the application repository to your cluster using GitOps. Your arc-cicd-demo-gitops GitOps repository contains the basic resources to get your app up and running on your arc-cicd-cluster cluster.

The initial GitOps repository contains only a manifest that creates the dev and stage namespaces corresponding to the deployment environments.

The GitOps connection that you create will automatically:

  • Sync the manifests in the manifest directory.
  • Update the cluster state.

The CI/CD workflow will populate the manifest directory with extra manifests to deploy the app.

  1. Create a new GitOps connection to your newly forked arc-cicd-demo-gitops repository in GitHub.

    az k8s-configuration flux create \
       --name cluster-config \
       --cluster-name arc-cicd-cluster \
       --namespace cluster-config \
       --resource-group myResourceGroup \
       -u  https://github.com/<Your organization>/arc-cicd-demo-gitops.git \
       --https-user <Azure Repos username> \
       --https-key <Azure Repos PAT token> \
       --scope cluster \
       --cluster-type managedClusters \
       --branch master \
       --kustomization name=cluster-config prune=true path=arc-cicd-cluster/manifests
    
  2. Check the state of the deployment in Azure portal.

    • If successful, you'll see both dev and stage namespaces created in your cluster.

Install GitOps Connector

  1. Add GitOps Connector repository to Helm repositories:
   helm repo add gitops-connector https://azure.github.io/gitops-connector/
  1. Install the connector to the cluster:
   helm upgrade -i gitops-connector gitops-connector/gitops-connector \
      --namespace flux-system \
      --set gitRepositoryType=GITHUB \
      --set ciCdOrchestratorType=GITHUB \
      --set gitOpsOperatorType=FLUX \
      --set gitHubGitOpsRepoName=arc-cicd-demo-src \
      --set gitHubGitOpsManifestsRepoName=arc-cicd-demo-gitops \
      --set gitHubOrgUrl=https://api.github.com/repos/<Your organization> \
      --set gitOpsAppURL=https://github.com/<Your organization>/arc-cicd-demo-gitops/commit \
      --set orchestratorPAT=<GitHub PAT token>
  1. Configure Flux to send notifications to GitOps connector:
cat <<EOF | kubectl apply -f -
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: gitops-connector
  namespace: flux-system
spec:
  eventSeverity: info
  eventSources:
  - kind: GitRepository
    name: cluster-config
  - kind: Kustomization
    name: cluster-config-cluster-config
  providerRef:
    name: gitops-connector
---
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Provider
metadata:
  name: gitops-connector
  namespace: flux-system
spec:
  type: generic
  address: http://gitops-connector:8080/gitopsphase
EOF

For the details on installation, refer to the GitOps Connector repository.

Create GitHub secrets

Create GitHub repository secrets

Secret Value
AZURE_CREDENTIALS Credentials for Azure in the following format {"clientId":"GUID","clientSecret":"GUID","subscriptionId":"GUID","tenantId":"GUID"}
AZURE_VOTE_IMAGE_REPO The full path to the Azure Vote App repository, for example azurearctest.azurecr.io/azvote
MANIFESTS_BRANCH master
MANIFESTS_FOLDER arc-cicd-cluster
MANIFESTS_REPO https://github.com/your-organization/arc-cicd-demo-gitops
VOTE_APP_TITLE Voting Application
AKS_RESOURCE_GROUP AKS Resource group. Needed for automated testing.
AKS_NAME AKS Name. Needed for automated testing.
PAT GitHub PAT token with the permission to PR to the GitOps repository

Create GitHub environment secrets

  1. Create az-vote-app-dev environment with the following secrets:
Secret Value
ENVIRONMENT_NAME Dev
TARGET_NAMESPACE dev
  1. Create az-vote-app-stage environment with the following secrets:
Secret Value
ENVIRONMENT_NAME Stage
TARGET_NAMESPACE stage

You're now ready to deploy to the dev and stage environments.

CI/CD Dev workflow

To start the CI/CD Dev workflow change the source code. In the application repository, update values in .azure-vote/src/azure-vote-front/config_file.cfg file and push the changes to the repository.

The CI/CD Dev workflow:

  • Ensures the application change passes all automated quality checks for deployment.
  • Does any extra validation that couldn't be completed in the PR pipeline.
  • Verifies the Docker image has changed and the new image is pushed.
  • Publishes the artifacts (Docker image tags, Manifest templates, Utils) that will be used by the following CD stages.
  • Deploys the application to Dev environment.
    • Generates manifests to the GitOps repository
    • Creates a PR to the GitOps repository for approval
  1. Find the PR created by the pipeline to the GitOps repository.

  2. Verify the changes to the GitOps repository. You should see:

    • High-level Helm template changes.
    • Low-level Kubernetes manifests that show the underlying changes to the desired state. Flux deploys these manifests.
  3. If everything looks good, approve and complete the PR.

  4. After a few minutes, Flux picks up the change and starts the deployment.

  5. Monitor the Git Commit status on the Commit history tab. Once it is succeeded the CD Stage workflow will start

  6. Forward the port locally using kubectl and ensure the app works correctly using:

    kubectl port-forward -n dev svc/azure-vote-front 8080:80
    
  7. View the Azure Vote app in your browser at http://localhost:8080/.

  8. Vote for your favorites and get ready to make some changes to the app.

CD Stage workflow

The CD Stage workflow starts automatically once Flux successfully deploys the application to dev environment and notifies GitHub actions via GitOps Connector.

The CD Stage workflow:

  • Runs application smoke tests against Dev environment
  • Deploys the application to Stage environment.
    • Generates manifests to the GitOps repository
    • Creates a PR to the GitOps repository for approval

Once the manifests PR to the Stage environment is merged and Flux successfully applied all the changes, it updates Git commit status in the GitOps repository.

Your deployment is now complete. This ends the CI/CD workflow. Refer to the GitHub GitOps Flow diagram in the application repository that explains in details the steps and techniques implemented in the CI/CD workflows used in this tutorial.

Clean up resources

If you're not going to continue to use this application, delete any resources with the following steps:

  1. Delete the Azure Arc GitOps configuration connection:

    az k8s-configuration flux delete \
          --name cluster-config \
          --cluster-name arc-cicd-cluster \
          --resource-group myResourceGroup \
          -t managedClusters --yes
    
  2. Delete GitOps Connector:

    helm uninstall gitops-connector -n flux-system
    kubectl delete alerts.notification.toolkit.fluxcd.io gitops-connector -n flux-system
    kubectl delete providers.notification.toolkit.fluxcd.io  gitops-connector -n flux-system
    

Next steps

In this tutorial, you have set up a full CI/CD workflow that implements DevOps from application development through deployment. Changes to the app automatically trigger validation and deployment, gated by manual approvals.

Advance to our conceptual article to learn more about GitOps and configurations with Azure Arc-enabled Kubernetes.