AKS appgw.ingress.k8s.io/v1/azureingressprohibitedtargets?allowWatchBookmarks=true ;: dial tcp 10.104.0.1:443: connect: connection refused

Question

I have AKS cluster with two address spaces:
10.103.0.0 - > for AKS resources
10.104.0.0 -> for AKS service

I am frequently getting connection refused error:
E0719 12:18:18.653126 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Service: Get "https://10.104.0.1:443/api/v1/services?resourceVersion=14962104": dial tcp 10.104.0.1:443: connect: connection refused
E0719 12:18:18.660431 1 reflector.go:380] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to watch *v1.AzureIngressProhibitedTarget: Get "https://10.104.0.1:443/apis/appgw.ingress.k8s.io/v1/azureingressprohibitedtargets?allowWatchBookmarks=true&resourceVersion=14962348&timeout=6m51s&timeoutSeconds=411&watch=true": dial tcp 10.104.0.1:443: connect: connection refused
E0719 12:18:18.699695 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Get "https://10.104.0.1:443/api/v1/endpoints?resourceVersion=14965055": dial tcp 10.104.0.1:443: connect: connection refused
E0719 12:18:25.959365 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Secret: Get "https://10.104.0.1:443/api/v1/secrets?resourceVersion=14962253": dial tcp 10.104.0.1:443: connect: connection refused
E0719 12:18:26.660252 1 reflector.go:380] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to watch *v1.AzureIngressProhibitedTarget: Get "https://10.104.0.1:443/apis/appgw.ingress.k8s.io/v1/azureingressprohibitedtargets?allowWatchBookmarks=true&resourceVersion=14962348&timeout=7m19s&timeoutSeconds=439&watch=true": dial tcp 10.104.0.1:443: connect: connection refused
E0719 12:18:26.911659 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.0.0-20200326020446-6240434e1ad6/tools/cache/reflector.go:125: Failed to list *v1.Pod: Get "https://10.104.0.1:443/api/v1/pods?resourceVersion=14962411": dial tcp 10.104.0.1:443: connect: connection refused

I am regularly receiving this connection refused error.