Azure Gitops helm controller broken (cannot list resource \"secrets\" in API group \"\" )

Question

We have a brand new azure managed AKS cluster where we activate azure gitops V2 via CLI:

az k8s-configuration flux create -g $RESOURCE_GROUP_NAME -c $CLUSTER_NAME -n gitops-infra --namespace gitops \
    -t managedClusters --scope cluster \
    -u ssh://git@ssh.dev.azure.com/v3/OUR-REPO --branch master \
    --ssh-private-key-file ~/.ssh/id_azure_git \
   --kustomization name=gitops path=./gitops/cluster/poc prune=true

Flux is created, but we can't install anything via Helm:

   helm-controller-754bff6895-j6cmh manager {"level":"error","ts":"2022-04-19T09:11:48.084Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"sentry","namespace":"sentry-poc","error":"failed to get last release revision: query: failed to query with labels: secrets is forbidden: User \"system:serviceaccount:sentry-poc:flux-applier\" cannot list resource \"secrets\" in API group \"\" in the namespace \"sentry-poc\": Azure does not have opinion for this user."}

Upon investigation with the flux team in their slack, we found out that the helm controller is started with two options that are missing in another cluster, we have (with working flux).

Excerpt from the helm controller manifest:

 containers:
    - name: manager
      image: mcr.microsoft.com/oss/fluxcd/helm-controller:v0.16.0
      args:
        - >-
          --events-addr=http://notification-controller.$(RUNTIME_NAMESPACE).svc.cluster.local/
        - '--watch-all-namespaces=true'
        - '--log-level=info'
        - '--log-encoding=json'
        - '--enable-leader-election'
        - '--no-cross-namespace-refs=true'
        - '--default-service-account=flux-applier'

The last two options are not present in the working version, I have removed these in the cluster in the helm controller deployment and the error message is gone and the helm charts are deployed.

Long story short:

Why are the new options there and how can I get rid of them via the CLI or alternatively, why are the rbac rules not correctly setup so that the service account setting works as expected?

Any hints?

Answer

Hi @Patrick Cornelissen (DPDHL IT Services), external ,
Thanks for reaching out to Microsoft QnA.
Can you check the below latest documentation on multi-tenancy feature : https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/tutorial-use-gitops-flux2#multi-tenancy.
The issue may be related.
Let me know if this helps? Meanwhile, I am trying to check more on this.

Edit :
I did check internally with the Team. Yes, this is related to the multi-tenancy fix that was recently released.

We don’t recommend that you remove those flags, since it means that the “namespaced” scoped configurations would not actually be namespaced (in the sense that anyone who has write-access to that namespace (even if they don’t have write access to other namespace) can actually deploy anything across the cluster).

Ideally, the workaround for this issue is to deploy your HelmRelease to the same namespace as the configuration so that the reconciliation all resides in that namespace with elevated privilege. This means you will have to set the “targetNamespace” on the HelmRelease to specify the namespace that you want to deploy the chart to.

Hope this helps.

-------------

Please do Accept as Answer and Upvote if you think the information provided was useful.

Azure Gitops helm controller broken (cannot list resource \"secrets\" in API group \"\" )

1 answer