How to support rolling restarts with AZ KeyVault csi driver

Question

I have a providerClass setup like so:

spec:
  provider: azure
  secretObjects:
    - secretName: keycloak-http
      type: Opaque
      data:
        - objectName: keycloak-password
          key: password
        - objectName: keycloak-user
          key: username
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: xxxx
    keyvaultName: xxx
    objects:  |
      array:
        - |
          objectName: keycloak-user
          objectType: secret              
          objectVersion: ""               
        - |
          objectName: keycloak-password
          objectType: secret
          objectVersion: ""
    tenantId: xxx

In my deployment, i have the volumes/mounts setup as per documentation, and have linked the secrets to env variables:

        - name: KEYCLOAK_ADMIN_USERNAME
          valueFrom:
            secretKeyRef:
              key: username
              name: keycloak-http
        - name: KEYCLOAK_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: keycloak-http
...
        volumeMounts:
        - mountPath: /secrets-store
          name: keycloak-inline-secrets
          readOnly: true
...
      volumes:
      - csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: azure-kv-keycloak-msi
        name: keycloak-inline-secrets

Now, the problem is that when i rotate this secret i can trigger a restart of the deployment, but it's configured with the rollingUpdate strategy, so it scales up a new pod, waits until it's running, and then deletes the old one.

This creates a scenario where the inline mount on /secrets-store has the correct, up to date secrets, but the env variables coming from the keycloak-http secret are still using the old version. I have found this happens because the k8s secret never gets updated in the case of a rolling restart - only if i scale the deploy manually to 0, then scale it back up does the k8s secret get rotated (since it is deleted with the pod, then recreated)

Is there a way around this, to force both the k8s secret and the mount to get updated properly?

Answer 1

Hello Vaibhav Dhawan,

Welcome to microsoft Q&A, Thankyou for posting your query here.

use an init container to fetch secrets directly from /secrets-store at pod startup and write them to a shared volume or a configuration file that your main application container can read.

This ensures that every new pod starts with the latest secrets:

initContainers:
- name: copy-keycloak-secrets
  image: alpine
  command: ['sh', '-c', 'cp /secrets-store/* /etc/secrets/']
  volumeMounts:
  - name: secrets-store
    mountPath: /secrets-store
  - name: secret-config
    mountPath: /etc/secrets

Hope this helps you.

If an answer has been helpful, please consider accepting the answer to help increase visibility of this question for other members of the Microsoft Q&A community. If not, please let us know what is still needed in the comments so the question can be answered. Thank you for helping to improve Microsoft Q&A!