Kubernetes Service to external services

Question

Kubernetes Service to external services

Roan van Heerden 0

I have a Kubernetes application that is running, in the application I am trying to enrich data via an external plug in (google translate). The service cannot connect to google and I am getting the below error:

 +- org.apache.http.conn.HttpHostConnectException: Connect to translation.googleapis.com:443 [translation.googleapis.com/172.217.169.10, translation.googleapis.com/2a00:1450:4009:823:0:0:0:200a] failed: Network is unreachable (connect failed)
  +- Connect to translation.googleapis.com:443 [translation.googleapis.com/172.217.169.10, translation.googleapis.com/2a00:1450:4009:823:0:0:0:200a] failed: Network is unreachable (connect failed)
   +- Network is unreachable (connect failed)

Cristian Gatjens 716 Reputation points Microsoft Employee

2023-05-16T21:57:24.3066667+00:00

Hello Roan,

I am assuming that you are seeing the error in the application pod that gets exposed by your service? Are you able to exec into the pod and curl the same enpoint?

What is the network path from the client to the endpoint?

I recommend the following link for basic connectivity troubleshooting steps:

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/troubleshoot-connection-pods-services-same-cluster

Please make sure you don't have any NSG's or a Firewall rule blocking the traffic to that endpoint.

Thanks!
Roan van Heerden 0 Reputation points

2023-05-17T08:10:59.8033333+00:00

Hey @Cristian Gatjens , thank you so much for the message.

I am very new to Kubernetes and this is my first app I have deployed like this. So I am accessing the application through the load balancer and it is a web based app. The application manages data and I am trying to use a google api enricher to translate some data.

So I am really not sure how to check the network path or how outbound traffic interacts with the application.

I also do not have a firewall set up.
Cristian Gatjens 716 Reputation points Microsoft Employee

2023-05-17T15:15:33.7966667+00:00

Hey Roan,

Is this the only app that you are exposing via a LB service? Do other apps work fine?

This could be an isolated problem with that Google API, there are some external sites describing similar behaviors:

https://stackoverflow.com/questions/28366402/failed-to-connect-to-www-googleapis-com-port-443-network-unreachable

https://stackoverflow.com/questions/33451107/failed-to-connect-to-www-googleapis-com-port-443-network-is-unreachable

In Kubernetes, when you use a LB service, that service uses endpoints that the service uses to forward the traffic to, you can check that by either describing the service "kubectl describe svc <service_name>" and checking the endpoints section or by running "kubectl get endpoints". Once you identify the endpoint pods, you can exec into them by running "kubectl exec -it <pod_name> -n <namespace_> -- bash or /bin/bash" and try a curl -vvv translation.googleapis.com to check if it responds as expected.
Roan van Heerden 0 Reputation points

2023-05-17T18:17:15.1233333+00:00

Hey @Cristian Gatjens , thank you so much for the message.

I am very new to Kubernetes and this is my first app I have deployed like this. So I am accessing the application through the load balancer and it is a web based app. The application manages data and I am trying to use a google api enricher to translate some data.

So I am really not sure how to check the network path or how outbound traffic interacts with the application.

I also do not have a firewall set up.

Roan van Heerden 0

Hey @Cristian Gatjens

Screenshot 2023-05-17 at 20.11.24

Screenshot 2023-05-17 at 20.11.41

Screenshot 2023-05-17 at 20.12.16

Thank you again for the help, I have followed your steps and get the below response:

I have also attached screenshots showing the different services that are routed via the load balancer

*   Trying 142.250.200.42:80...
* Connected to translation.googleapis.com (142.250.200.42) port 80 (#0)
> GET / HTTP/1.1
> Host: translation.googleapis.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Date: Wed, 17 May 2023 18:15:56 GMT
< Content-Type: text/html; charset=UTF-8
< Server: ESF
< Content-Length: 1561
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< 
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/</code> was not found on this server.  <ins>That’s all we know.</ins>
* Connection #0 to host translation.googleapis.com left intact

Roan van Heerden 0 Reputation points

2023-05-23T10:00:17+00:00

Hi @Cristian Gatjens have you got any help on the above?
Cristian Gatjens 716 Reputation points Microsoft Employee

2023-05-23T16:10:30.0666667+00:00

@Anonymous everything looks good from the outputs you shared. I forgot to ask if you are using an ingress controller like NGINX to route traffic? Something like:

https://learn.microsoft.com/en-us/azure/aks/ingress-basic?tabs=azure-cli

You can try creating the test apps and checking if you are able to connect, if so, maybe this is an issue with the Google endpoint and we should involve them.

Are you using a specific deployment for that service that you can share the link with me so I can try replicating this behavior?

Thanks!

Roan van Heerden 0

Hey @Cristian Gatjens ,

Yes I am using a NGINX controller. I have attached the YAML file for you.

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
  uid: 24d49316-4f9d-47e3-b35b-e53e0e679e0c
  resourceVersion: '5570732'
  creationTimestamp: '2023-04-11T14:49:50Z'
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.1.2
    helm.sh/chart: ingress-nginx-4.0.18
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.1.2","helm.sh/chart":"ingress-nginx-4.0.18"},"name":"ingress-nginx-controller","namespace":"ingress-nginx"},"spec":{"externalTrafficPolicy":"Local","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"appProtocol":"https","name":"https","port":443,"protocol":"TCP","targetPort":"https"}],"selector":{"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx"},"type":"LoadBalancer"}}
  finalizers:
    - service.kubernetes.io/load-balancer-cleanup
  managedFields:
    - manager: kubectl-client-side-apply
      operation: Update
      apiVersion: v1
      time: '2023-04-11T14:49:50Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/component: {}
            f:app.kubernetes.io/instance: {}
            f:app.kubernetes.io/managed-by: {}
            f:app.kubernetes.io/name: {}
            f:app.kubernetes.io/part-of: {}
            f:app.kubernetes.io/version: {}
            f:helm.sh/chart: {}
        f:spec:
          f:allocateLoadBalancerNodePorts: {}
          f:externalTrafficPolicy: {}
          f:internalTrafficPolicy: {}
          f:ipFamilies: {}
          f:ipFamilyPolicy: {}
          f:ports:
            .: {}
            k:{"port":443,"protocol":"TCP"}:
              .: {}
              f:appProtocol: {}
              f:name: {}
              f:port: {}
              f:protocol: {}
              f:targetPort: {}
          f:selector: {}
          f:sessionAffinity: {}
          f:type: {}
    - manager: cloud-controller-manager
      operation: Update
      apiVersion: v1
      time: '2023-04-11T14:49:59Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:finalizers:
            .: {}
            v:"service.kubernetes.io/load-balancer-cleanup": {}
        f:status:
          f:loadBalancer:
            f:ingress: {}
      subresource: status
    - manager: Mozilla
      operation: Update
      apiVersion: v1
      time: '2023-04-26T11:58:52Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            f:kubectl.kubernetes.io/last-applied-configuration: {}
spec:
  ports:
    - name: https
      protocol: TCP
      appProtocol: https
      port: 443
      targetPort: https
      nodePort: 32142
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  clusterIP: 10.0.154.66
  clusterIPs:
    - 10.0.154.66
  type: LoadBalancer
  sessionAffinity: None
  externalTrafficPolicy: Local
  healthCheckNodePort: 31784
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  allocateLoadBalancerNodePorts: true
  internalTrafficPolicy: Cluster
status:
  loadBalancer:
    ingress:
      - ip: 20.58.112.111

I am installing Semarchy on Kubernetes, please see the link here: https://www.semarchy.com/doc/semarchy-xdm/xdm/5.3/Install/azure/aks/install-on-aks.html

So the app works fine, it is just when I try to enrich data that it fails.

How can I test outbound endpoints through the load balancer?

Also please see attached my ingress controller file


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "sticksemarchy"
    nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
    nginx.ingress.kubernetes.io/proxy-body-size: 30m
  name: ingress
  namespace: default
spec:
  rules:
  - host: semarchy.passive.912048bebc6c438ea019.ukwest.aksapp.io
    http:
      paths:
      - backend:
          service:
            name: semarchy-appserver-passive
            port:
              number: 8080
        path: /
        pathType: Prefix
  - host: semarchy.active.912048bebc6c438ea019.ukwest.aksapp.io
    http:
      paths:
      - backend:
          service:
            name: semarchy-appserver-active
            port:
              number: 8080
        path: /
        pathType: Prefix

Please also see attached a screenshot of the deployed application.

Screenshot 2023-05-23 at 19.03.08

Cristian Gatjens 716 Reputation points Microsoft Employee

2023-05-23T21:57:10.39+00:00

Roan,

I would directly curl the LB service that is exposing your app.

You can create a test pod and install curl package.

curl -vvv <LB_IP>:Port

Reference: https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/connection-issues-application-hosted-aks-cluster#step-2-check-whether-the-application-is-reachable-from-the-service

So the app works fine, it is just when I try to enrich data that it fails.

I am not sure about that statement - what does "enrich data" mean?

Please check the ingress controller pod logs when hitting the endpoints to confirm that we are redirecting the traffic correctly - https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/connection-issues-application-hosted-aks-cluster#scenarios-that-use-an-ingress-instead-of-a-service

Example command: kubectl logs ingress-nginx-controller -n ingress-nginx
Roan van Heerden 0 Reputation points

2023-05-24T13:16:27.54+00:00

Hey @Cristian Gatjens

I tried:

curl -vvv <LB_IP>:Port

And got this response:

root@aks-ssh:/# curl -vvv 20.58.112.111:443

* Trying 20.58.112.111:443...

* Connected to 20.58.112.111 (20.58.112.111) port 443 (#0)

> GET / HTTP/1.1

> Host: 20.58.112.111:443

> User-Agent: curl/7.74.0

> Accept: */*

>

* Mark bundle as not supporting multiuse

< HTTP/1.1 400 Bad Request

< Date: Wed, 24 May 2023 13:14:32 GMT

< Content-Type: text/html

< Content-Length: 248

< Connection: close

<

<html>

<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>

<body>

<center><h1>400 Bad Request</h1></center>

<center>The plain HTTP request was sent to HTTPS port</center>

<hr><center>nginx</center>

</body>

</html>

* Closing connection 0
Roan van Heerden 0 Reputation points

2023-05-24T13:23:11.29+00:00

@Cristian Gatjens ,

I also did the following:

root@aks-ssh:/# curl -Iv http://20.58.112.111:443

* Trying 20.58.112.111:443...

* Connected to 20.58.112.111 (20.58.112.111) port 443 (#0)

> HEAD / HTTP/1.1

> Host: 20.58.112.111:443

> User-Agent: curl/7.74.0

> Accept: */*

>

* Mark bundle as not supporting multiuse

< HTTP/1.1 400 Bad Request

HTTP/1.1 400 Bad Request

< Date: Wed, 24 May 2023 13:19:12 GMT

Date: Wed, 24 May 2023 13:19:12 GMT

< Content-Type: text/html

Content-Type: text/html

< Content-Length: 248

Content-Length: 248

< Connection: close

Connection: close

And this:

root@aks-ssh:/# curl -Iv https://semarchy.active.912048bebc6c438ea019.ukwest.aksapp.io/semarchy/welcome

* Trying 20.58.112.111:443...

* Connected to semarchy.active.912048bebc6c438ea019.ukwest.aksapp.io (20.58.112.111) port 443 (#0)

* ALPN, offering h2

* ALPN, offering http/1.1

* successfully set certificate verify locations:

* CAfile: /etc/ssl/certs/ca-certificates.crt

* CApath: /etc/ssl/certs

* TLSv1.3 (OUT), TLS handshake, Client hello (1):

* TLSv1.3 (IN), TLS handshake, Server hello (2):

* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):

* TLSv1.3 (IN), TLS handshake, Certificate (11):

* TLSv1.3 (OUT), TLS alert, unknown CA (560):

* SSL certificate problem: self signed certificate

* Closing connection 0

curl: (60) SSL certificate problem: self signed certificate

More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not

establish a secure connection to it. To learn more about this situation and

how to fix it, please visit the web page mentioned above.
Roan van Heerden 0 Reputation points

2023-05-25T13:24:11.85+00:00

Hi @Cristian Gatjens is it possible to have a call to look at resolving this?

Your answer

Cristian Gatjens 716 Reputation points Microsoft Employee

2023-05-16T21:57:24.3066667+00:00

Hello Roan,

I am assuming that you are seeing the error in the application pod that gets exposed by your service? Are you able to exec into the pod and curl the same enpoint?

What is the network path from the client to the endpoint?

I recommend the following link for basic connectivity troubleshooting steps:

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/troubleshoot-connection-pods-services-same-cluster

Please make sure you don't have any NSG's or a Firewall rule blocking the traffic to that endpoint.

Thanks!
Roan van Heerden 0 Reputation points

2023-05-17T08:10:59.8033333+00:00

Hey @Cristian Gatjens , thank you so much for the message.

I am very new to Kubernetes and this is my first app I have deployed like this. So I am accessing the application through the load balancer and it is a web based app. The application manages data and I am trying to use a google api enricher to translate some data.

So I am really not sure how to check the network path or how outbound traffic interacts with the application.

I also do not have a firewall set up.
Cristian Gatjens 716 Reputation points Microsoft Employee

2023-05-17T15:15:33.7966667+00:00

Hey Roan,

Is this the only app that you are exposing via a LB service? Do other apps work fine?

This could be an isolated problem with that Google API, there are some external sites describing similar behaviors:

https://stackoverflow.com/questions/28366402/failed-to-connect-to-www-googleapis-com-port-443-network-unreachable

https://stackoverflow.com/questions/33451107/failed-to-connect-to-www-googleapis-com-port-443-network-is-unreachable

In Kubernetes, when you use a LB service, that service uses endpoints that the service uses to forward the traffic to, you can check that by either describing the service "kubectl describe svc <service_name>" and checking the endpoints section or by running "kubectl get endpoints". Once you identify the endpoint pods, you can exec into them by running "kubectl exec -it <pod_name> -n <namespace_> -- bash or /bin/bash" and try a curl -vvv translation.googleapis.com to check if it responds as expected.
Roan van Heerden 0 Reputation points

2023-05-17T18:17:15.1233333+00:00

Hey @Cristian Gatjens , thank you so much for the message.

I am very new to Kubernetes and this is my first app I have deployed like this. So I am accessing the application through the load balancer and it is a web based app. The application manages data and I am trying to use a google api enricher to translate some data.

So I am really not sure how to check the network path or how outbound traffic interacts with the application.

I also do not have a firewall set up.
Roan van Heerden 0 Reputation points

2023-05-23T10:00:17+00:00

Hi @Cristian Gatjens have you got any help on the above?
Cristian Gatjens 716 Reputation points Microsoft Employee

2023-05-23T16:10:30.0666667+00:00

@Anonymous everything looks good from the outputs you shared. I forgot to ask if you are using an ingress controller like NGINX to route traffic? Something like:

https://learn.microsoft.com/en-us/azure/aks/ingress-basic?tabs=azure-cli

You can try creating the test apps and checking if you are able to connect, if so, maybe this is an issue with the Google endpoint and we should involve them.

Are you using a specific deployment for that service that you can share the link with me so I can try replicating this behavior?

Thanks!
Cristian Gatjens 716 Reputation points Microsoft Employee

2023-05-23T21:57:10.39+00:00

Roan,

I would directly curl the LB service that is exposing your app.

You can create a test pod and install curl package.

curl -vvv <LB_IP>:Port

Reference: https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/connection-issues-application-hosted-aks-cluster#step-2-check-whether-the-application-is-reachable-from-the-service

So the app works fine, it is just when I try to enrich data that it fails.

I am not sure about that statement - what does "enrich data" mean?

Please check the ingress controller pod logs when hitting the endpoints to confirm that we are redirecting the traffic correctly - https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/connection-issues-application-hosted-aks-cluster#scenarios-that-use-an-ingress-instead-of-a-service

Example command: kubectl logs ingress-nginx-controller -n ingress-nginx
Roan van Heerden 0 Reputation points

2023-05-24T13:16:27.54+00:00

Hey @Cristian Gatjens

I tried:

curl -vvv <LB_IP>:Port

And got this response:

root@aks-ssh:/# curl -vvv 20.58.112.111:443

* Trying 20.58.112.111:443...

* Connected to 20.58.112.111 (20.58.112.111) port 443 (#0)

> GET / HTTP/1.1

> Host: 20.58.112.111:443

> User-Agent: curl/7.74.0

> Accept: */*

>

* Mark bundle as not supporting multiuse

< HTTP/1.1 400 Bad Request

< Date: Wed, 24 May 2023 13:14:32 GMT

< Content-Type: text/html

< Content-Length: 248

< Connection: close

<

<html>

<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>

<body>

<center><h1>400 Bad Request</h1></center>

<center>The plain HTTP request was sent to HTTPS port</center>

<hr><center>nginx</center>

</body>

</html>

* Closing connection 0
Roan van Heerden 0 Reputation points

2023-05-24T13:23:11.29+00:00

@Cristian Gatjens ,

I also did the following:

root@aks-ssh:/# curl -Iv http://20.58.112.111:443

* Trying 20.58.112.111:443...

* Connected to 20.58.112.111 (20.58.112.111) port 443 (#0)

> HEAD / HTTP/1.1

> Host: 20.58.112.111:443

> User-Agent: curl/7.74.0

> Accept: */*

>

* Mark bundle as not supporting multiuse

< HTTP/1.1 400 Bad Request

HTTP/1.1 400 Bad Request

< Date: Wed, 24 May 2023 13:19:12 GMT

Date: Wed, 24 May 2023 13:19:12 GMT

< Content-Type: text/html

Content-Type: text/html

< Content-Length: 248

Content-Length: 248

< Connection: close

Connection: close

And this:

root@aks-ssh:/# curl -Iv https://semarchy.active.912048bebc6c438ea019.ukwest.aksapp.io/semarchy/welcome

* Trying 20.58.112.111:443...

* Connected to semarchy.active.912048bebc6c438ea019.ukwest.aksapp.io (20.58.112.111) port 443 (#0)

* ALPN, offering h2

* ALPN, offering http/1.1

* successfully set certificate verify locations:

* CAfile: /etc/ssl/certs/ca-certificates.crt

* CApath: /etc/ssl/certs

* TLSv1.3 (OUT), TLS handshake, Client hello (1):

* TLSv1.3 (IN), TLS handshake, Server hello (2):

* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):

* TLSv1.3 (IN), TLS handshake, Certificate (11):

* TLSv1.3 (OUT), TLS alert, unknown CA (560):

* SSL certificate problem: self signed certificate

* Closing connection 0

curl: (60) SSL certificate problem: self signed certificate

More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not

establish a secure connection to it. To learn more about this situation and

how to fix it, please visit the web page mentioned above.
Roan van Heerden 0 Reputation points

2023-05-25T13:24:11.85+00:00

Hi @Cristian Gatjens is it possible to have a call to look at resolving this?

Share via

Kubernetes Service to external services

Your answer