Arc enabled kubernetes - Envoy looses ? certificate ? after cluster restart and return connection reset on listeners

Question

Relate to my original question (more logs and installation command there) https://stackoverflow.com/questions/68544812/arc-enabled-kubernetes-envoy-looses-certificate-after-cluster-restart-and
and https://github.com/envoyproxy/envoy/issues/17484

connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: CERTIFICATE_VERIFY_FAILED  

after first deployment, before cluster restart

few minutes after the cluster restart, when 'something' sync

THE PROBLEM
You deploy arc-enabled k8s, connect the location, enable extension, all works fine, you can access your appservice via URL, nice!
Then you restart the cluster, pods are recreated, and within few minutes, all breaks.
Based on my observation, either

• app-controller sync (or does not sync) certificate/token with Azure,
• OR envoy has problem mounting XDR certificates provided by the app-service pod (they share volume)
• OR maybe service account token is not refreshed?

I guess, as this is not clear to me what is missing, I thought it's some secret, but did not catch any...
but this is strange

"envoy","msg":"error reading default cert","error":"140261821150336:error:0D06B08E:asn1 encoding routines:asn1_d2i_read_bio:not enough data

There is litte docs on Arc :/ for an open project I'd love to see the source helm charts
I've noticed you store one in the cluster secret, yet I failed to decrypt, any suggestion there?

kubectl get secret sh.helm.release.v1.appservice-ext-node1-v1.v3   --namespace appservice-ns-node1-v1  -o=jsonpath={.data.release}   |base64 -d > /tmp/helm.base  

Why envoy gives the ERROR_CONN_RESET:

https://www.envoyproxy.io/docs/envoy/latest/configuration/security/secret#config-secret-discovery-service

If a listener server certificate needs to be fetched by SDS remotely,
it will NOT be marked as active, its port will not be opened before
the certificates are fetched. If Envoy fails to fetch the certificates
due to connection failures, or bad response data, the listener will be
marked as active, and the port will be open, but the connection to the
port will be reset.

Most interesting error, found in appservice-ext-node1-v1-k8se-app-controller-85cb587976-nwr6h

{"level":"info","ts":1627301876.563232,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"127.0.0.1:8080"}  
{"level":"error","ts":1627301876.6502664,"logger":"envoy","msg":"Error reading default cert","error":"140453565756544:error:0D06B08E:asn1 encoding routines:asn1_d2i_read_bio:not enough data:../crypto/asn1/a_d2i_fp.c:198:\n\nexit status 1","stacktrace":"main.main\n\t/__w/k4a  
pps/k4apps/cmd/appcontroller/main.go:124\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"}  
{"level":"error","ts":1627301876.6528075,"logger":"envoy","msg":"error reading dapr cert","error":"secrets "dapr-trust-bundle" not found","stacktrace":"main.main\n\t/__w/k4apps/k4apps/cmd/appcontroller/main.go:124\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"}  
{"level":"info","ts":1627301876.7418113,"logger":"setup","msg":"starting manager"}  
I0726 12:17:56.741905 1 leaderelection.go:243] attempting to acquire leader lease appservice-ns-node1-v1/appservice-ns-node1-v1-appservice-ns-node1-v1...  
{"level":"info","ts":1627301876.7421775,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}  
{"level":"error","ts":1627301877.841797,"logger":"envoy","msg":"error reading default cert","error":"140261821150336:error:0D06B08E:asn1 encoding routines:asn1_d2i_read_bio:not enough data:../crypto/asn1/a_d2i_fp.c:198:\n\nexit status 1","stacktrace":"github.com/microsoft/k  
4apps/pkg/envoy.(*XDSManagementServer).updateTLSCert\n\t/__w/k4apps/k4apps/pkg/envoy/envoy.go:379\ngithub.com/microsoft/k4apps/pkg/envoy.(*XDSManagementServer).watchDefaultTLSCert.func3\n\t/__w/k4apps/k4apps/pkg/envoy/envoy.go:336\nk8s.io/client-go/tools/cache.ResourceEvent  
HandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/controller.go:231\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/shared_informer.go:777\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.f  
unc1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.5/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.5/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v  
0.20.5/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.5/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/shared_informer.go:771\nk  
8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.5/pkg/util/wait/wait.go:73"}  
{"level":"info","ts":1627301877.8429492,"logger":"envoy","msg":"Processed on startup","count":2}  
{"level":"info","ts":1627301877.8429906,"logger":"envoy.stopwatch","msg":"measured: ","Initializing snapshot":1101}  
{"level":"info","ts":1627301877.8430026,"logger":"envoy","msg":"starting xds and auth server on port 9090"}  
I0726 12:18:13.706604 1 leaderelection.go:253] successfully acquired lease appservice-ns-node1-v1/appservice-ns-node1-v1-appservice-ns-node1-v1  

Answer 1

Installation script: https://pastebin.com/igCc5KwR

Answer 2

BTW, are the helm charts or azure-arc components available publicly? is it an Open source? or accessible for customers ?