Schedule a pod only if kafka pods are ready

Question

Hi we have implemeted a kafka cluster using the open source operator called strimzi https://strimzi.io/docs/operators/latest/using.html

in our AKS cluster. we are daily starting/stopping our AKS cluster due to budget constraints. But the problem is that when we start our AKS cluster the consumer pod is getting scheduled first and is already in running state even before our kafka pods are up. And the consumer pod is trying to establish connection to kafka pods to consume messages, resulting in failure. so is there any concept/configuration w.r.t kubernetes to delay the scheduling of our consumer pod or any other concept in which we control our consumer pod to not be in ready state until the kafka pods are up.
please help

Thanks in advance

Answer 1

@Shreyas Arani , thank you for your question.

This sounds like a textbook use case for init containers.

Init containers: specialized containers that run before app containers in a Pod. Init containers can contain utilities or setup scripts not present in an app image.

A Pod can have multiple containers running apps within it, but it can also have one or more init containers, which are run before the app containers are started.

Init containers are exactly like regular containers, except:

• Init containers always run to completion.
• Each init container must complete successfully before the next one starts.

If a Pod's init container fails, the kubelet repeatedly restarts that init container until it succeeds. However, if the Pod has a restartPolicy of Never, and an init container fails during startup of that Pod, Kubernetes treats the overall Pod as failed.

To specify an init container for a Pod, add the initContainers field into the Pod specification, as an array of container items (similar to the app containers field and its contents). See Container in the API reference for more details.

The status of the init containers is returned in .status.initContainerStatuses field as an array of the container statuses (similar to the .status.containerStatuses field).

Here are a few examples.

I would recommend you to add an init container in your consumer pod spec which will wait for the Kafka service to be Ready, holding the consumer pod at PodInitializing state. Once the Kafka pod(s) are Ready, the init container will complete execution and then your consumer app container will be started.

----
Hope this helps.

Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.