Cannot access prometheus and grafana from browser

Question

Cannot access prometheus and grafana from browser

Sandeep Yarashi 1

I have aks,appgw , sample webapp ,public ip deployed in azure. In aks I installed prometheus and grafana using helm.
I can curl -L http:localhost.com:9090 (prometheus port -9090) in aks but can't access localhost:9090 in browser as my laptop and aks is in different network. I want to access prometheus and grafana with public IP of app gateway.
For eg:public ip is 10.0.2.2 I want to access prometheus by 10.0.2.2/prometheus and
access grafana by 10.0.2.2/ grafana. How to achieve this.

Sebastian Pacheco 371 Reputation points

2022-08-23T18:04:42.617+00:00

hello @Sandeep Yarashi ...

sorry for my english, when you install grafana from helm the access is only from localhost by default.

In your notebook (linux) install the kubectl client so that you have access to Azure, for example, then in the console of your notebook do the query to the namespace of prometheus "kubectl get pods -n prometheus",

then in your notebook issue the command:

kubectl port-forward -n prometheus prometheus-grafana-68db696c47-jn5x5 3000

(prometheus-grafana-68db696c47-jn5x5 is the pod that is running)

if all goes well, in your notebook browser enter localhost:3000

with that you will be able to enter grafana from your notebook... I hope it will be of help.

1 answer

Your answer

Sebastian Pacheco 371 Reputation points

2022-08-23T18:04:42.617+00:00

hello @Sandeep Yarashi ...

sorry for my english, when you install grafana from helm the access is only from localhost by default.

In your notebook (linux) install the kubectl client so that you have access to Azure, for example, then in the console of your notebook do the query to the namespace of prometheus "kubectl get pods -n prometheus",

then in your notebook issue the command:

kubectl port-forward -n prometheus prometheus-grafana-68db696c47-jn5x5 3000

(prometheus-grafana-68db696c47-jn5x5 is the pod that is running)

if all goes well, in your notebook browser enter localhost:3000

with that you will be able to enter grafana from your notebook... I hope it will be of help.

Answer 1

SRIJIT-BOSE-MSFT 4,346 Microsoft Employee Moderator

@Sandeep Yarashi , Thank you for your question.

This sounds like a perfect use case of an ingress controller (in your case it would be best to use an Application Gateway ingress Controller) for path based routing of prometheus and grafana services running on the AKS cluster.

The ingress controller must be deployed before installing grafana and prometheus.

For Grafana Helm values must have:

grafana:  
  ingress:  
    enabled: true  
    annotations:  
      kubernetes.io/ingress.class: "nginx"  
      nginx.ingress.kubernetes.io/rewrite-target: /  
      nginx.ingress.kubernetes.io/backend-protocol: "HTTP"  
    paths:  
    - /grafana

For prometheus Helm values must have:

prometheus:  
  enabled: true  
  service:  
    nodePort: 9090  
    type: NodePort  
  ingress:  
    enabled: true  
    annotations:  
      kubernetes.io/ingress.class: nginx  
      nginx.ingress.kubernetes.io/rewrite-target: /  
      nginx.ingress.kubernetes.io/backend-protocol: "HTTP"  
    paths:  
    - /prometheus

Note:

I can curl -L http:localhost.com:9090 (prometheus port -9090) in aks but can't access localhost:9090 in browser as my laptop and aks is in different network.

The kubectl proxy:

runs on a user's desktop or in a pod
proxies from a localhost address to the Kubernetes apiserver
client to proxy uses HTTP
proxy to apiserver uses HTTPS
locates apiserver
adds authentication headers

Without a public IP address this can be used to access services running on the cluster from your workstation machine which is in a different network.

Hope this helps.

Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.

Sandeep Yarashi 1 Reputation point

2021-08-16T13:25:44.263+00:00

Can you explain this in a step by step process to make it work. I am unable to get it working.
SRIJIT-BOSE-MSFT 4,346 Reputation points Microsoft Employee Moderator

2021-08-16T20:57:02.41+00:00
@Sandeep Yarashi , the Helm values I mentioned earlier would suffice for a basic ingress controller. Thank you for choosing Application Gateway Ingress Controller. I missed adding the following information in my earlier answer. For an Ingress resource to be observed by AGIC, it must be annotated with kubernetes.io/ingress.class: azure/application-gateway. Only then AGIC will work with the Ingress resource in question. [Reference]

Deploy AGIC on an AKS cluster via AKS add-on (either Greenfield or Brownfield)
[[contd in next comment]]

SRIJIT-BOSE-MSFT 4,346 Microsoft Employee Moderator

@Sandeep Yarashi ,

Create a grafana-values.yaml with:

grafana:  
   ingress:  
     enabled: true  
     annotations:  
       kubernetes.io/ingress.class: azure/application-gateway  
       kubernetes.io/ingress.class: "nginx"  
       nginx.ingress.kubernetes.io/rewrite-target: /  
       appgw.ingress.kubernetes.io/backend-protocol: http  
     paths:  
     - /grafana

Create a prometheus-values.yaml with:

prometheus:  
   enabled: true  
   service:  
     nodePort: 9090  
     type: NodePort  
   ingress:  
     enabled: true  
     annotations:  
       kubernetes.io/ingress.class: azure/application-gateway  
       kubernetes.io/ingress.class: nginx  
       nginx.ingress.kubernetes.io/rewrite-target: /  
       appgw.ingress.kubernetes.io/backend-protocol: http  
     paths:  
     - /prometheus

Install grafana and prometheus helm charts respectively as follows:

helm install -n <namespace> install -f grafana-values.yaml <grafana-installation-name> <grafana-helm-repository>
helm install -n <namespace> install -f prometheus-values.yaml <prometheus-installation-name> <prometehus-helm-repository>

Hope this helps. Please do let us know if you are facing any challenges.

Share via

Cannot access prometheus and grafana from browser

1 answer

Your answer