本文介紹如何在 Azure Kubernetes 服務 (AKS) 上配置和部署 Valkey 集群,包括建立 Valkey 集群 ConfigMap、主集群 Pod 和副本集群 Pod 以確保冗餘和區域複製,以及 Pod 中斷預算 (PDB) 以確保高可用性。
附註
本文提到的術語 master (即主伺服器) 和slave (即從伺服器) 已不再是 Microsoft 使用的術語。 從 Valkey 軟體移除字詞時,我們會將其從本文中移除。
連線到 AKS 叢集
附註
如果你用的是 Terraform,部署 基礎設施時,務必把程式碼中的佔位符替換成 Terraform 指令的實際輸出。
設定
kubectl使用az aks get-credentials命令連線至 AKS 叢集。az aks get-credentials --resource-group $MY_RESOURCE_GROUP_NAME --name $MY_CLUSTER_NAME --overwrite-existing --output table
建立命名空間
使用
kubectl create namespace命令建立 Valkey 叢集的命名空間。kubectl create namespace valkey --dry-run=client --output yaml | kubectl apply -f -範例輸出︰
namespace/valkey created
建立祕密
使用 openssl 產生 Valkey 叢集的隨機密碼,並使用
az keyvault secret set命令將它儲存在 Azure 金鑰保存庫中。export SECRET=$(openssl rand -base64 32) echo requirepass $SECRET > /tmp/valkey-password-file.conf echo primaryauth $SECRET >> /tmp/valkey-password-file.conf az keyvault secret set --vault-name $MY_KEYVAULT_NAME --name valkey-password-file --file /tmp/valkey-password-file.conf --output none rm /tmp/valkey-password-file.conf設定原則,以允許使用者指派的身分識別使用
az keyvault set-policy命令取得祕密。az keyvault set-policy --name $MY_KEYVAULT_NAME --object-id $userAssignedObjectID --secret-permissions get --output table
取得使用者指派的身份資訊
使用
az aks show指令來取得由 Azure Key Vault Secrets Provider 外掛程式建立的身份 ID 和物件 ID。export userAssignedIdentityID=$(az aks show --resource-group $MY_RESOURCE_GROUP_NAME --name $MY_CLUSTER_NAME --query addonProfiles.azureKeyvaultSecretsProvider.identity.clientId --output tsv) export userAssignedObjectID=$(az aks show --resource-group $MY_RESOURCE_GROUP_NAME --name $MY_CLUSTER_NAME --query addonProfiles.azureKeyvaultSecretsProvider.identity.objectId --output tsv)
設定存取儲存在 Azure Key Vault 的 Valkey 密碼
使用
SecretProviderClass命令建立kubectl apply資源,以存取儲存在金鑰保存庫中的 Valkey 密碼。export TENANT_ID=$(az account show --query tenantId --output tsv) kubectl apply -f - <<EOF --- apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: valkey-password namespace: valkey spec: provider: azure parameters: usePodIdentity: "false" useVMManagedIdentity: "true" userAssignedIdentityID: "${userAssignedIdentityID}" keyvaultName: ${MY_KEYVAULT_NAME} # the name of the AKV instance objects: | array: - | objectName: valkey-password-file objectAlias: valkey-password-file.conf objectType: secret tenantId: "${TENANT_ID}" # the tenant ID of the AKV instance EOF
建立 Valkey 組態檔
建立資源
ConfigMap來儲存 Valkey 組態檔。kubectl apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: valkey-cluster namespace: valkey data: valkey.conf: |+ cluster-enabled yes cluster-node-timeout 15000 cluster-config-file /data/nodes.conf appendonly yes protected-mode yes dir /data port 6379 include /etc/valkey-password/valkey-password-file.conf EOF範例輸出︰
configmap/valkey-cluster created
建立 Valkey 主要叢集 Pod
建立
StatefulSet資源,其spec.affinity目標是使用kubectl apply命令,讓區域 1 和區域 2 中的所有主要複本最好位於不同的節點中。kubectl apply -f - <<EOF --- apiVersion: apps/v1 kind: StatefulSet metadata: name: valkey-masters namespace: valkey spec: serviceName: "valkey-masters" replicas: 3 selector: matchLabels: app: valkey template: metadata: labels: app: valkey appCluster: valkey-masters spec: terminationGracePeriodSeconds: 20 affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: agentpool operator: In values: - valkey - key: topology.kubernetes.io/zone operator: In values: - ${MY_LOCATION}-1 - matchExpressions: - key: agentpool operator: In values: - valkey - key: topology.kubernetes.io/zone operator: In values: - ${MY_LOCATION}-2 podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - valkey topologyKey: topology.kubernetes.io/zone - weight: 90 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - valkey topologyKey: kubernetes.io/hostname containers: - name: role-master-checker image: "${MY_ACR_REGISTRY}.azurecr.io/valkey:latest" command: - "/bin/bash" - "-c" args: [ "while true; do role=\$(valkey-cli --pass \$(cat /etc/valkey-password/valkey-password-file.conf | awk '{print \$2; exit}') role | awk '{print \$1; exit}'); if [ \"\$role\" = \"slave\" ]; then valkey-cli --pass \$(cat /etc/valkey-password/valkey-password-file.conf | awk '{print \$2; exit}') cluster failover; fi; sleep 30; done" ] volumeMounts: - name: valkey-password mountPath: /etc/valkey-password readOnly: true - name: valkey image: "${MY_ACR_REGISTRY}.azurecr.io/valkey:latest" env: - name: VALKEY_PASSWORD_FILE value: "/etc/valkey-password/valkey-password-file.conf" - name: MY_POD_IP valueFrom: fieldRef: fieldPath: status.podIP command: - "valkey-server" args: - "/conf/valkey.conf" - "--cluster-announce-ip" - "\$(MY_POD_IP)" resources: requests: cpu: "100m" memory: "100Mi" ports: - name: valkey containerPort: 6379 protocol: "TCP" - name: cluster containerPort: 16379 protocol: "TCP" volumeMounts: - name: conf mountPath: /conf readOnly: false - name: data mountPath: /data readOnly: false - name: valkey-password mountPath: /etc/valkey-password readOnly: true volumes: - name: valkey-password csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: valkey-password - name: conf configMap: name: valkey-cluster defaultMode: 0755 volumeClaimTemplates: - metadata: name: data spec: accessModes: [ "ReadWriteOnce" ] storageClassName: managed-csi-premium resources: requests: storage: 20Gi EOF範例輸出︰
statefulset.apps/valkey-masters created
建立 Valkey 複本叢集 Pod
為 Valkey 複製品建立第二個
StatefulSet資源,spec.affinity目標是讓所有複製品都保持在第 3 區,最好是不同節點,並使用kubectl apply命令。kubectl apply -f - <<EOF --- apiVersion: apps/v1 kind: StatefulSet metadata: name: valkey-replicas namespace: valkey spec: serviceName: "valkey-replicas" replicas: 3 selector: matchLabels: app: valkey template: metadata: labels: app: valkey appCluster: valkey-replicas spec: terminationGracePeriodSeconds: 20 affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: agentpool operator: In values: - valkey - key: topology.kubernetes.io/zone operator: In values: - ${MY_LOCATION}-3 podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 90 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - valkey topologyKey: kubernetes.io/hostname containers: - name: valkey image: "${MY_ACR_REGISTRY}.azurecr.io/valkey:latest" env: - name: VALKEY_PASSWORD_FILE value: "/etc/valkey-password/valkey-password-file.conf" - name: MY_POD_IP valueFrom: fieldRef: fieldPath: status.podIP command: - "valkey-server" args: - "/conf/valkey.conf" - "--cluster-announce-ip" - "\$(MY_POD_IP)" resources: requests: cpu: "100m" memory: "100Mi" ports: - name: valkey containerPort: 6379 protocol: "TCP" - name: cluster containerPort: 16379 protocol: "TCP" volumeMounts: - name: conf mountPath: /conf readOnly: false - name: data mountPath: /data readOnly: false - name: valkey-password mountPath: /etc/valkey-password readOnly: true volumes: - name: valkey-password csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: valkey-password - name: conf configMap: name: valkey-cluster defaultMode: 0755 volumeClaimTemplates: - metadata: name: data spec: accessModes: [ "ReadWriteOnce" ] storageClassName: managed-csi-premium resources: requests: storage: 20Gi EOF範例輸出︰
statefulset.apps/valkey-replicas created
驗證網繭和節點分佈
使用 和
master-Nreplica-N命令驗證 和kubectl get nodeskubectl get pods是否在不同的節點和區域中執行。kubectl get pods -n valkey -o wide kubectl get node -o custom-columns=Name:.metadata.name,Zone:".metadata.labels.topology\.kubernetes\.io/zone"範例輸出︰
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES valkey-masters-0 1/1 Running 0 2m55s 10.224.0.4 aks-valkey-18693609-vmss000004 <none> <none> valkey-masters-1 1/1 Running 0 2m31s 10.224.0.137 aks-valkey-18693609-vmss000000 <none> <none> valkey-masters-2 1/1 Running 0 2m7s 10.224.0.222 aks-valkey-18693609-vmss000001 <none> <none> valkey-replicas-0 1/1 Running 0 88s 10.224.0.237 aks-valkey-18693609-vmss000005 <none> <none> valkey-replicas-1 1/1 Running 0 70s 10.224.0.18 aks-valkey-18693609-vmss000002 <none> <none> valkey-replicas-2 1/1 Running 0 48s 10.224.0.242 aks-valkey-18693609-vmss000005 <none> <none> Name Zone aks-nodepool1-17621399-vmss000000 centralus-1 aks-nodepool1-17621399-vmss000001 centralus-2 aks-nodepool1-17621399-vmss000003 centralus-3 aks-valkey-18693609-vmss000000 centralus-1 aks-valkey-18693609-vmss000001 centralus-2 aks-valkey-18693609-vmss000002 centralus-3 aks-valkey-18693609-vmss000003 centralus-1 aks-valkey-18693609-vmss000004 centralus-2 aks-valkey-18693609-vmss000005 centralus-3等待大約三分鐘,讓所有
master-N和replica-NPod 執行,然後再繼續下一步。
建立 Headless 服務
建立三個無
Service頭資源(第一個給整個叢集,第二個給主節點,第三個給複本),用這個指令取得 Valkey Podkubectl apply的 IP 位址。kubectl apply -f - <<EOF apiVersion: v1 kind: Service metadata: name: valkey-cluster namespace: valkey spec: clusterIP: None ports: - name: valkey-port port: 6379 protocol: TCP targetPort: 6379 selector: app: valkey sessionAffinity: None type: ClusterIP EOF kubectl apply -f - <<EOF apiVersion: v1 kind: Service metadata: name: valkey-masters namespace: valkey spec: clusterIP: None ports: - name: valkey-port port: 6379 protocol: TCP targetPort: 6379 selector: app: valkey appCluster: valkey-masters sessionAffinity: None type: ClusterIP EOF kubectl apply -f - <<EOF apiVersion: v1 kind: Service metadata: name: valkey-replicas namespace: valkey spec: clusterIP: None ports: - name: valkey-port port: 6379 protocol: TCP targetPort: 6379 selector: app: valkey appCluster: valkey-replicas sessionAffinity: None type: ClusterIP EOF範例輸出︰
service/valkey-cluster created service/valkey-masters created service/valkey-replicas created
建立 Pod 中斷預算 (PDB)
建立 Pod 中斷預算 (PDB) 以確保在自發性中斷期間 (例如升級或維護) 最多只有一個 Pod 無法使用。 這有助於維護 Kubernetes 叢集中 Valkey 應用程式的穩定性和可用性。
kubectl apply -f - <<EOF apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: valkey namespace: valkey spec: maxUnavailable: 1 selector: matchLabels: app: valkey EOF範例輸出︰
poddisruptionbudget.policy/valkey created
執行 Valkey 叢集
在 Valkey 叢集中,插槽分配是資料在節點之間分配的基本部分。 Valkey 叢集(類似於 Redis 叢集)將金鑰空間劃分為 16,384 個雜湊槽,這些雜湊槽均勻分佈在叢集中的主節點上。
在以下章節中,我們建立了一個由三個節點組成的 Valkey 叢集,包含三個副本,確保:
- 插槽全覆蓋(共 16,384 個插槽)
- 透過複寫提供高可用性。
- 適當的角色指派和叢集健康情況驗證。
使用區域 1 和 2 中的主要節點初始化 Valkey 叢集,並設定插槽分佈
在區域 1 和 2 中建立具有三個主要節點的叢集,並使用命令
kubectl exec將所有 16,384 個雜湊插槽平均分佈在它們之間。kubectl exec -it -n valkey valkey-masters-0 -- valkey-cli --cluster create --cluster-yes --cluster-replicas 0 \ valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379 \ valkey-masters-1.valkey-masters.valkey.svc.cluster.local:6379 \ valkey-masters-2.valkey-masters.valkey.svc.cluster.local:6379 \ --pass ${SECRET}範例輸出︰
>>> Performing hash slots allocation on 3 nodes... Master[0] -> Slots 0 - 5460 Master[1] -> Slots 5461 - 10922 Master[2] -> Slots 10923 - 16383 M: ee6ac1d00d3f016b6f46c7ce11199bc1a7809a35 valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379 slots:[0-5460] (5461 slots) master M: fd1fb98db83976478e05edd3d2a02f9a13badd80 valkey-masters-1.valkey-masters.valkey.svc.cluster.local:6379 slots:[5461-10922] (5462 slots) master M: ea47bf57ae7080ef03164a4d48b662c7b4c8770e valkey-masters-2.valkey-masters.valkey.svc.cluster.local:6379 slots:[10923-16383] (5461 slots) master >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join ... >>> Performing Cluster Check (using node valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379) M: ee6ac1d00d3f016b6f46c7ce11199bc1a7809a35 valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379 slots:[0-5460] (5461 slots) master M: ea47bf57ae7080ef03164a4d48b662c7b4c8770e 10.224.0.176:6379 slots:[10923-16383] (5461 slots) master M: fd1fb98db83976478e05edd3d2a02f9a13badd80 10.224.0.247:6379 slots:[5461-10922] (5462 slots) master [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
在區域 3 中新增 Valkey 複本以啟用高可用性
使用下列
kubectl exec命令,在不同區域中為每個主要節點新增一個複本,以確保容錯和自動容錯移轉功能。kubectl exec -ti -n valkey valkey-masters-0 -- valkey-cli --cluster add-node \ valkey-replicas-0.valkey-replicas.valkey.svc.cluster.local:6379 \ valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379 --cluster-slave \ --pass ${SECRET} kubectl exec -ti -n valkey valkey-masters-0 -- valkey-cli --cluster add-node \ valkey-replicas-1.valkey-replicas.valkey.svc.cluster.local:6379 \ valkey-masters-1.valkey-masters.valkey.svc.cluster.local:6379 --cluster-slave \ --pass ${SECRET} kubectl exec -ti -n valkey valkey-masters-0 -- valkey-cli --cluster add-node \ valkey-replicas-2.valkey-replicas.valkey.svc.cluster.local:6379 \ valkey-masters-2.valkey-masters.valkey.svc.cluster.local:6379 --cluster-slave \ --pass ${SECRET}範例輸出︰
>>> Adding node valkey-replicas-0.valkey-replicas.valkey.svc.cluster.local:6379 to cluster valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379 >>> Performing Cluster Check (using node valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379) M: ee6ac1d00d3f016b6f46c7ce11199bc1a7809a35 valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379 slots:[0-5460] (5461 slots) master M: ea47bf57ae7080ef03164a4d48b662c7b4c8770e 10.224.0.176:6379 slots:[10923-16383] (5461 slots) master M: fd1fb98db83976478e05edd3d2a02f9a13badd80 10.224.0.247:6379 slots:[5461-10922] (5462 slots) master [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. Automatically selected master valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379 >>> Send CLUSTER MEET to node valkey-replicas-0.valkey-replicas.valkey.svc.cluster.local:6379 to make it join the cluster. Waiting for the cluster to join >>> Configure node as replica of valkey-masters-0.valkey-masters.valkey.svc.cluster.local:6379. [OK] New node added correctly. >>> Adding node valkey-replicas-1.valkey-replicas.valkey.svc.cluster.local:6379 to cluster valkey-masters-1.valkey-masters.valkey.svc.cluster.local:6379 >>> Performing Cluster Check (using node valkey-masters-1.valkey-masters.valkey.svc.cluster.local:6379) M: fd1fb98db83976478e05edd3d2a02f9a13badd80 valkey-masters-1.valkey-masters.valkey.svc.cluster.local:6379 slots:[5461-10922] (5462 slots) master S: 0ebceb60cbcc31da9040159440a1f4856b992907 10.224.0.224:6379 slots: (0 slots) slave replicates ee6ac1d00d3f016b6f46c7ce11199bc1a7809a35 M: ea47bf57ae7080ef03164a4d48b662c7b4c8770e 10.224.0.176:6379 slots:[10923-16383] (5461 slots) master M: ee6ac1d00d3f016b6f46c7ce11199bc1a7809a35 10.224.0.14:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. Automatically selected master valkey-masters-1.valkey-masters.valkey.svc.cluster.local:6379 >>> Send CLUSTER MEET to node valkey-replicas-1.valkey-replicas.valkey.svc.cluster.local:6379 to make it join the cluster. Waiting for the cluster to join >>> Configure node as replica of valkey-masters-1.valkey-masters.valkey.svc.cluster.local:6379. [OK] New node added correctly. >>> Adding node valkey-replicas-2.valkey-replicas.valkey.svc.cluster.local:6379 to cluster valkey-masters-2.valkey-masters.valkey.svc.cluster.local:6379 >>> Performing Cluster Check (using node valkey-masters-2.valkey-masters.valkey.svc.cluster.local:6379) M: ea47bf57ae7080ef03164a4d48b662c7b4c8770e valkey-masters-2.valkey-masters.valkey.svc.cluster.local:6379 slots:[10923-16383] (5461 slots) master S: 0ebceb60cbcc31da9040159440a1f4856b992907 10.224.0.224:6379 slots: (0 slots) slave replicates ee6ac1d00d3f016b6f46c7ce11199bc1a7809a35 S: fa44edff683e2e01ee5c87233f9f3bc35c205dce 10.224.0.103:6379 slots: (0 slots) slave replicates fd1fb98db83976478e05edd3d2a02f9a13badd80 M: ee6ac1d00d3f016b6f46c7ce11199bc1a7809a35 10.224.0.14:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) M: fd1fb98db83976478e05edd3d2a02f9a13badd80 10.224.0.247:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. Automatically selected master valkey-masters-2.valkey-masters.valkey.svc.cluster.local:6379 >>> Send CLUSTER MEET to node valkey-replicas-2.valkey-replicas.valkey.svc.cluster.local:6379 to make it join the cluster. Waiting for the cluster to join >>> Configure node as replica of valkey-masters-2.valkey-masters.valkey.svc.cluster.local:6379. [OK] New node added correctly.
驗證容器的角色和複製狀態
確認每個主要和複本都已正確設定,並使用下列
kubectl exec命令建立複寫關係。for x in $(seq 0 2); do echo "valkey-masters-$x"; kubectl exec -n valkey valkey-masters-$x -- valkey-cli --pass ${SECRET} role; echo; done for x in $(seq 0 2); do echo "valkey-replicas-$x"; kubectl exec -n valkey valkey-replicas-$x -- valkey-cli --pass ${SECRET} role; echo; done範例輸出︰
valkey-masters-0 master 84 10.224.0.224 6379 84 valkey-masters-1 master 84 10.224.0.103 6379 84 valkey-masters-2 master 70 10.224.0.200 6379 70 valkey-replicas-0 slave 10.224.0.14 6379 connected 98 valkey-replicas-1 slave 10.224.0.247 6379 connected 98 valkey-replicas-2 slave 10.224.0.176 6379 connected 84
後續步驟
若要深入了解如何在 Azure Kubernetes Service 上部署開放原始碼的軟體 (AKS),請參閱下列文章:
參與者
此文章由 Microsoft 維護。 最初是由下列參與者所撰寫:
- Nelly Kiboi | 服務工程師
- Saverio Proto | 首席客戶體驗工程師
- Don High | 首席客戶工程師
- LaBrina Loving | 首席服務工程師
- Ken Kilty | 首席 TPM
- Russell de Pina | 首席 TPM
- Colin Mixon | 產品經理
- Ketan Chawda | 資深客戶工程師
- Naveed Kharadi | 客戶體驗工程師
- Erin Schaffer |內容開發人員 2