Liveness probe - breaks port redirection on node

Question

Liveness probe - breaks port redirection on node

SfDen 141

I have standalone cluster 7.1.417.9590 on Windows Server 2019 and containerized application.
I add Liveness probe in ApplicationManifest - https://learn.microsoft.com/en-us/azure/service-fabric/probes-codepackage
<CodePackagePolicy CodePackageRef="Code">
<Probes>
<Probe Type="Liveness" FailureThreshold="3" SuccessThreshold="2" InitialDelaySeconds="10" PeriodSeconds="5" TimeoutSeconds="5">
<HttpGet Path="/healthcheck" Port="2532" Scheme="http"/>
</Probe>
</Probes>
</CodePackagePolicy>
Application successfully upgrade, but http endpoint becomes unavailable. I tested this on several applications.

prmanhas-MSFT 17,976 Reputation points Microsoft Employee Moderator

2020-07-09T08:25:18.037+00:00

@SfDen Have you had a chance to look at the report for your container

I’m suspecting that tour probe isn’t working which is causing your container to restart making it’s normal http endpoint unavailable

Try to use another probe type maybe using TCP, etc.
SfDen 141 Reputation points

2020-07-09T09:05:02.997+00:00

Health State my container - OK.
Http Liveness probe works properly. If i'm manually, from container, closed http endpoint, then container restarted, as expected.
Container endpoint available:
Test-NetConnection -ComputerName <ContainerIP> -port 2532
True

Node endpoint unavailable:
Test-NetConnection -ComputerName <NodeIP> -port 2532
False
Without Liveness probe all it`s OK
prmanhas-MSFT 17,976 Reputation points Microsoft Employee Moderator

2020-07-13T10:56:43.647+00:00

@SfDen Thank you for your response.

Can you help me understand what is the current issue you are facing?
SfDen 141 Reputation points

2020-07-14T10:02:31.847+00:00

The problem is described in detail above.
What details are missing?
I have a containerized service accessible via endpoint "http://<NodeIP>:2532".
If I add Http Liveness probe, then endpoint "http://<NodeIP>:2532", becomes unavailable. I can’t access my service.
All I do is add in ApplicationManifest:
<CodePackagePolicy CodePackageRef="Code">
<Probes>
<Probe Type="Liveness" FailureThreshold="3" SuccessThreshold="2" InitialDelaySeconds="10" PeriodSeconds="5" TimeoutSeconds="5">
<HttpGet Path="/healthcheck" Port="2532" Scheme="http"/>
</Probe>
</Probes>
</CodePackagePolicy>
prmanhas-MSFT 17,976 Reputation points Microsoft Employee Moderator

2020-07-17T06:27:30.307+00:00

@SfDen The above configuration would check for /healthcheck at port 2532, I am assuming this is the port application listens to and /healthcheck is the endpoint which return http 200 status code. If that is not the case, change the configuration.

Apart from it, this configuration would wait for 25 seconds from the time container is started to ensure application becomes healthy otherwise the container is restarted as mentioned in the documentation (when failure threshold is reached, container is restarted).

Please check if application starts listening and responding to http call during this time. I would suggest to increase the Initial Delay Seconds and trying to see if that helps. Sometime application takes time to start and due to liveness probe, container keeps on restarting causing the application to never start.

Hope this helps.
SfDen 141 Reputation points

2020-07-17T10:31:01.277+00:00

For example:
In the attachment is an elementary ServiceManifest and ApplicationManifest for the package.
Is the app running on your test cluster?
12726-applicationmanifest.xml

12833-servicemanifest.xml
prmanhas-MSFT 17,976 Reputation points Microsoft Employee Moderator

2020-07-21T11:20:03.013+00:00

@SfDen-4602 The issue seems to require in depth troubleshooting, so what I will suggest is to open a Support Ticket with our Microsoft Technical Support who will be having the required expertise to help you out on the issue.

If you don't have the ability to open a technical support ticket, you can email me with Subject as "ATTN: Preeti" at AzCommunity@microsoft.com and provide me with your Subscription ID and link to this thread. I can then enable your subscription for one time free support.

Also, once you get the issue fixed, request you to reply back here on the thread with the resolution steps for the benefit of the community.

If the suggested response helped you resolve your issue, do click on "Mark as Answer" and "Up-Vote" for the answer that helped you for benefit of the community.

Thanks.
prmanhas-MSFT 17,976 Reputation points Microsoft Employee Moderator

2020-07-23T13:27:50.41+00:00

@SfDen Can you please confirm if you need any further help from my side?

Please 'Accept as answer' if it helped, so that it can help others in the community looking for help on similar topics