Edit

Share via

Kubernetes hosting

Kubernetes is a popular choice for hosting Orleans applications. Orleans runs in Kubernetes without specific configuration; however, it can also take advantage of extra knowledge the hosting platform provides.

The Microsoft.Orleans.Hosting.Kubernetes package adds integration for hosting an Orleans application in a Kubernetes cluster. The package provides an extension method, UseKubernetesHosting, performing the following actions:

Note that the Kubernetes hosting package doesn't use Kubernetes for clustering. A separate clustering provider is still needed. For more information on configuring clustering, see the Server configuration documentation.

This functionality imposes some requirements on service deployment:

  • Silo names must match pod names.
  • Pods must have orleans/serviceId and orleans/clusterId labels corresponding to the silo's ServiceId and ClusterId. The UseKubernetesHosting method propagates these labels into the corresponding Orleans options from environment variables.
  • Pods must have the following environment variables set: POD_NAME, POD_NAMESPACE, POD_IP, ORLEANS_SERVICE_ID, ORLEANS_CLUSTER_ID.

The following example shows how to configure these labels and environment variables correctly:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dictionary-app
  labels:
    orleans/serviceId: dictionary-app
spec:
  selector:
    matchLabels:
      orleans/serviceId: dictionary-app
  replicas: 3
  template:
    metadata:
      labels:
        # This label identifies the service to Orleans
        orleans/serviceId: dictionary-app

        # This label identifies an instance of a cluster to Orleans.
        # Typically, this is the same value as the previous label, or any
        # fixed value.
        # In cases where you don't use rolling deployments (for example,
        # blue/green deployments),
        # this value can allow for distinct clusters that don't communicate
        # directly with each other,
        # but still share the same storage and other resources.
        orleans/clusterId: dictionary-app
    spec:
      containers:
        - name: main
          image: my-registry.azurecr.io/my-image
          imagePullPolicy: Always
          ports:
          # Define the ports Orleans uses
          - containerPort: 11111
          - containerPort: 30000
          env:
          # The Azure Storage connection string for clustering is injected as an
          # environment variable.
          # You must create it separately using a command such as:
          # > kubectl create secret generic az-storage-acct `
          #     --from-file=key=./az-storage-acct.txt
          - name: STORAGE_CONNECTION_STRING
            valueFrom:
              secretKeyRef:
                name: az-storage-acct
                key: key
          # Configure settings to let Orleans know which cluster it belongs to
          # and which pod it's running in.
          - name: ORLEANS_SERVICE_ID
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['orleans/serviceId']
          - name: ORLEANS_CLUSTER_ID
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['orleans/clusterId']
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: DOTNET_SHUTDOWNTIMEOUTSECONDS
            value: "120"
          request:
            # Set resource requests
      terminationGracePeriodSeconds: 180
      imagePullSecrets:
        - name: my-image-pull-secret
  minReadySeconds: 60
  strategy:
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1

For RBAC-enabled clusters, granting the required access to the Kubernetes service account for the pods might also be necessary:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: orleans-hosting
rules:
- apiGroups: [ "" ]
  resources: ["pods"]
  verbs: ["get", "watch", "list", "delete", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: orleans-hosting-binding
subjects:
- kind: ServiceAccount
  name: default
  apiGroup: ''
roleRef:
  kind: Role
  name: orleans-hosting
  apiGroup: ''

Liveness, readiness, and startup probes

Kubernetes can probe pods to determine service health. For more information, see Configure Liveness, Readiness and Startup Probes in the Kubernetes documentation.

Orleans uses a cluster membership protocol to promptly detect and recover from process or network failures. Each node monitors a subset of other nodes, sending periodic probes. If a node fails to respond to multiple successive probes from multiple other nodes, the cluster forcibly removes it. Once a failed node learns of its removal, it terminates immediately. Kubernetes restarts the terminated process, which then attempts to rejoin the cluster.

Kubernetes probes help determine whether a process in a pod is executing and isn't stuck in a zombie state. These probes don't verify inter-pod connectivity or responsiveness, nor do they perform application-level functionality checks. If a pod fails to respond to a liveness probe, Kubernetes might eventually terminate that pod and reschedule it. Kubernetes probes and Orleans probes are therefore complementary.

The recommended approach is configuring Liveness Probes in Kubernetes that perform a simple, local-only check that the application performs as intended. These probes serve to terminate the process if there's a total freeze, for example, due to a runtime fault or another unlikely event.

Resource quotas

Kubernetes works with the operating system to implement resource quotas. This allows enforcing CPU and memory reservations and/or limits. For a primary application serving interactive load, implementing restrictive limits isn't recommended unless necessary. It's important to note that requests and limits differ substantially in meaning and implementation location. Before setting requests or limits, take time to gain a detailed understanding of how they are implemented and enforced. For example, memory might not be measured uniformly between Kubernetes, the Linux kernel, and the monitoring system. CPU quotas might not be enforced as expected.

Troubleshooting

Pods crash, complaining that KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined

Full exception message:

Unhandled exception. k8s.Exceptions.KubeConfigException: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
at k8s.KubernetesClientConfiguration.InClusterConfig()
  • Check that the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT environment variables are set inside the Pod. Check by executing the command kubectl exec -it <pod_name> /bin/bash -c env.
  • Ensure automountServiceAccountToken is set to true in the Kubernetes deployment.yaml. For more information, see Configure Service Accounts for Pods.