AGIC ingress on AKS unabled to populate status.loadbalancer.ingress when in two different resource group
Hello Microsoft.
I'm trying to create a structure within my organization where I use the same virtual network for all AKS clusters so that if need be they can communicate security and locally inside a Virtual Network within Azure.
My Issue:
Information:
Resource group = "rg"
VNet, X, in rg vnet_rg
AGIC, Y, in rg appgw_rg
AKS, Z, in rg cluster_rg
Node pool J in rg NodeResourceGroup
Load balancer with external IP, in NodeResourceGroup
My Kubernetes cluster is deployed through Argo CD with my application on a different cluster than where the actual Ingress resource is located (on a different cluster).
Based on my previous deployment, this works when the gateway, VNet, and load balancer are in the same VNet. However, I want to avoid using the same resource group for multiple different resources in Azure to create a better structure and overview.
When I then deploy an application with the necessary things I can get my application properly exposed to internet and actual connect to it, but when the ingress is deployed it's in a never ending progressing state because it can't get the loadbalancer IP.
What my question is, is there any where for me to tell the ingress deployment in k8s where to find or locate the loadbalancer? When it was all the same rg, the ingress deployment would populate these credentials it self, but now when it not populated, im unsure if there actually are loadbalancing for my k8s cluster apps through the gateway.
Azure Kubernetes Service
-
Anonymous
2025-02-12T07:21:05.22+00:00 Hi Morten Lund Jakobsen,
Thank you for reaching out to us on the Microsoft Q&A forum.
Based on your query i understand that you are trying to place services in different resources groups so Agic unable to populate the load balancer IP.
Could you please describe the status of your Ingress for more detailed logs:kubectl describe ingress <ingress-name> -n <namespace>
If you have any further queries, please let us know in the comment.
-
Morten Lund Jakobsen • 0 Reputation points
2025-02-12T07:35:04.6433333+00:00 Name: app-qa Labels: app.kubernetes.io/instance=app-qa app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=app app.kubernetes.io/version=1.16.0 argocd.argoproj.io/instance=app-qa helm.sh/chart=app-0.1.0 Namespace: creditro Address: Ingress Class: <none> Default backend: <default> TLS: qa-app-tls terminates app-qa.cloud.creditro.com Rules: Host Path Backends ---- ---- -------- app-qa.cloud.creditro.com / app-qa:3000 (10.80.0.28:3000) Annotations: acme.cert-manager.io/dns01-edit-in-place: true appgw.ingress.kubernetes.io/backend-path-prefix: / appgw.ingress.kubernetes.io/health-probe-path: / appgw.ingress.kubernetes.io/ssl-redirect: true appgw.ingress.kubernetes.io/use-private-ip: false cert-manager.io/acme-challenge-type: dns01 cert-manager.io/cluster-issuer: letsencrypt-prod kubernetes.io/ingress.class: azure/application-gateway Events: <none>
-
Anonymous
2025-02-13T06:49:01.1666667+00:00 Hi @Morten Lund Jakobsen,
In the above output i didn't find any ingress class name we need more details from your end to know issue can you please provide output of below commands:
Please check the status and logs of all AGIC pods by using below command:
kubectl get pods -n <agic-namespace>
kubectl logs <agic-pod-name> -n <agic-namespace>
Thank you.
-
Morten Lund Jakobsen • 0 Reputation points
2025-02-13T08:01:18.03+00:00 kube describe the ingress-azure pod:
Name: ingress-azure-85d86d9fbd-5g89b Namespace: default Priority: 0 Service Account: ingress-azure Node: aks-systempool-41479419-vmss000000/10.80.0.4 Start Time: Tue, 11 Feb 2025 11:01:35 +0100 Labels: app=ingress-azure azure.workload.identity/use=true pod-template-hash=85d86d9fbd release=ingress-azure Annotations: checksum/config: a0b7e4711bbb73909543116e5ebc8cef42ccdc5dba379bc78fcdf86c63a145cc prometheus.io/port: 8123 prometheus.io/scrape: true Status: Running IP: 10.80.0.7 IPs: IP: 10.80.0.7 Controlled By: ReplicaSet/ingress-azure-85d86d9fbd Containers: ingress-azure: Container ID: containerd://4a2c922c21b2b1f74ad0e6671ea6cb1e20ae410ff7375f950d62b9e8b56d62c1 Image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.7 Image ID: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:a1c3a7e43bf141aaec0f60f5c096450ed84063f0f1b9e337f9bf753eb98894d6 Port: <none> Host Port: <none> State: Running Started: Tue, 11 Feb 2025 11:01:36 +0100 Ready: True Restart Count: 0 Liveness: http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3 Readiness: http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: ingress-azure ConfigMap Optional: false Environment: AZURE_CLOUD_PROVIDER_LOCATION: /etc/appgw/azure.json AGIC_POD_NAME: ingress-azure-85d86d9fbd-5g89b (v1:metadata.name) AGIC_POD_NAMESPACE: default (v1:metadata.namespace) AZURE_CLIENT_ID: ece434be-a056-47b0-87af-7730a765e61c AZURE_TENANT_ID: 860d0fbb-99e3-4e1c-815d-0b83992e928c AZURE_FEDERATED_TOKEN_FILE: /var/run/secrets/azure/tokens/azure-identity-token AZURE_AUTHORITY_HOST: https://login.microsoftonline.com/ Mounts: /etc/appgw/ from azure (ro) /var/run/secrets/azure/tokens from azure-identity-token (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vvwhm (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: azure: Type: HostPath (bare host directory volume) Path: /etc/kubernetes/ HostPathType: Directory kube-api-access-vvwhm: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true azure-identity-token: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3600 QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: <none>
k logs
I0211 14:25:00.156746 1 mutate_app_gateway.go:163] Generated config: { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev", -- App Gwy config -- "location": "northeurope", -- App Gwy config -- "properties": { -- App Gwy config -- "backendAddressPools": [ -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/backendAddressPools/pool-creditro-app-qa-3000-bp-3000", -- App Gwy config -- "name": "pool-creditro-app-qa-3000-bp-3000", -- App Gwy config -- "properties": { -- App Gwy config -- "backendAddresses": [ -- App Gwy config -- { -- App Gwy config -- "ipAddress": "10.80.0.28" -- App Gwy config -- } -- App Gwy config -- ] -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- ], -- App Gwy config -- "backendHttpSettingsCollection": [ -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/backendHttpSettingsCollection/bp-creditro-app-qa-3000-3000-app-qa", -- App Gwy config -- "name": "bp-creditro-app-qa-3000-3000-app-qa", -- App Gwy config -- "properties": { -- App Gwy config -- "cookieBasedAffinity": "Disabled", -- App Gwy config -- "path": "/", -- App Gwy config -- "pickHostNameFromBackendAddress": false, -- App Gwy config -- "port": 3000, -- App Gwy config -- "probe": { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/probes/pb-creditro-app-qa-3000-app-qa" -- App Gwy config -- }, -- App Gwy config -- "protocol": "Http", -- App Gwy config -- "requestTimeout": 30 -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- ], -- App Gwy config -- "customErrorConfigurations": [], -- App Gwy config -- "enableHttp2": true, -- App Gwy config -- "frontendIPConfigurations": [ -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/frontendIPConfigurations/creditro-dev-frontend-ip", -- App Gwy config -- "name": "creditro-dev-frontend-ip", -- App Gwy config -- "properties": { -- App Gwy config -- "privateIPAllocationMethod": "Dynamic", -- App Gwy config -- "privateLinkConfiguration": { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/privateLinkConfigurations/pls-connection" -- App Gwy config -- }, -- App Gwy config -- "publicIPAddress": { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/publicIPAddresses/appgw-ip" -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- ], -- App Gwy config -- "frontendPorts": [ -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/frontendPorts/creditro-dev-frontend-port", -- App Gwy config -- "name": "creditro-dev-frontend-port", -- App Gwy config -- "properties": { -- App Gwy config -- "port": 80 -- App Gwy config -- } -- App Gwy config -- }, -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/frontendPorts/fp-443", -- App Gwy config -- "name": "fp-443", -- App Gwy config -- "properties": { -- App Gwy config -- "port": 443 -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- ], -- App Gwy config -- "gatewayIPConfigurations": [ -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/gatewayIPConfigurations/appGatewayIpConfig", -- App Gwy config -- "name": "appGatewayIpConfig", -- App Gwy config -- "properties": { -- App Gwy config -- "subnet": { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/creditro-vnet-rg/providers/Microsoft.Network/virtualNetworks/creditro-vnet/subnets/appgw-subnet-dev" -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- ], -- App Gwy config -- "httpListeners": [ -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/httpListeners/fl-29c2c5d7eab3c14afbe36ff1c2c75948", -- App Gwy config -- "name": "fl-29c2c5d7eab3c14afbe36ff1c2c75948", -- App Gwy config -- "properties": { -- App Gwy config -- "frontendIPConfiguration": { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/frontendIPConfigurations/creditro-dev-frontend-ip" -- App Gwy config -- }, -- App Gwy config -- "frontendPort": { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/frontendPorts/creditro-dev-frontend-port" -- App Gwy config -- }, -- App Gwy config -- "hostNames": [ -- App Gwy config -- "app-qa.cloud.creditro.com" -- App Gwy config -- ], -- App Gwy config -- "protocol": "Http", -- App Gwy config -- "requireServerNameIndication": false -- App Gwy config -- } -- App Gwy config -- }, -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/httpListeners/fl-a487ec67d2d4268de9806b8c1288c6bd", -- App Gwy config -- "name": "fl-a487ec67d2d4268de9806b8c1288c6bd", -- App Gwy config -- "properties": { -- App Gwy config -- "frontendIPConfiguration": { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/frontendIPConfigurations/creditro-dev-frontend-ip" -- App Gwy config -- }, -- App Gwy config -- "frontendPort": { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/frontendPorts/fp-443" -- App Gwy config -- }, -- App Gwy config -- "hostNames": [ -- App Gwy config -- "app-qa.cloud.creditro.com" -- App Gwy config -- ], -- App Gwy config -- "protocol": "Https", -- App Gwy config -- "requireServerNameIndication": false, -- App Gwy config -- "sslCertificate": { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/sslCertificates/cert-creditro-qa-app-tls" -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- ], -- App Gwy config -- "loadDistributionPolicies": [], -- App Gwy config -- "privateLinkConfigurations": [ -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/privateLinkConfigurations/pls-connection", -- App Gwy config -- "name": "pls-connection", -- App Gwy config -- "properties": { -- App Gwy config -- "ipConfigurations": [ -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/privateLinkConfigurations/pls-connection/ipConfigurations/privateLinkIpConfig1", -- App Gwy config -- "name": "privateLinkIpConfig1", -- App Gwy config -- "properties": { -- App Gwy config -- "primary": false, -- App Gwy config -- "privateIPAllocationMethod": "Dynamic", -- App Gwy config -- "subnet": { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/creditro-vnet-rg/providers/Microsoft.Network/virtualNetworks/creditro-vnet/subnets/aks-subnet-dev" -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- ] -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- ], -- App Gwy config -- "probes": [ -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/probes/pb-creditro-app-qa-3000-app-qa", -- App Gwy config -- "name": "pb-creditro-app-qa-3000-app-qa", -- App Gwy config -- "properties": { -- App Gwy config -- "host": "app-qa.cloud.creditro.com", -- App Gwy config -- "interval": 30, -- App Gwy config -- "match": {}, -- App Gwy config -- "minServers": 0, -- App Gwy config -- "path": "/", -- App Gwy config -- "pickHostNameFromBackendHttpSettings": false, -- App Gwy config -- "protocol": "Http", -- App Gwy config -- "timeout": 30, -- App Gwy config -- "unhealthyThreshold": 3 -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- ], -- App Gwy config -- "redirectConfigurations": [ -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/redirectConfigurations/sslr-fl-a487ec67d2d4268de9806b8c1288c6bd", -- App Gwy config -- "name": "sslr-fl-a487ec67d2d4268de9806b8c1288c6bd", -- App Gwy config -- "properties": { -- App Gwy config -- "includePath": true, -- App Gwy config -- "includeQueryString": true, -- App Gwy config -- "redirectType": "Permanent", -- App Gwy config -- "targetListener": { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/httpListeners/fl-a487ec67d2d4268de9806b8c1288c6bd" -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- ], -- App Gwy config -- "requestRoutingRules": [ -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/requestRoutingRules/rr-29c2c5d7eab3c14afbe36ff1c2c75948", -- App Gwy config -- "name": "rr-29c2c5d7eab3c14afbe36ff1c2c75948", -- App Gwy config -- "properties": { -- App Gwy config -- "httpListener": { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/httpListeners/fl-29c2c5d7eab3c14afbe36ff1c2c75948" -- App Gwy config -- }, -- App Gwy config -- "priority": 19000, -- App Gwy config -- "redirectConfiguration": { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/redirectConfigurations/sslr-fl-a487ec67d2d4268de9806b8c1288c6bd" -- App Gwy config -- }, -- App Gwy config -- "ruleType": "Basic" -- App Gwy config -- } -- App Gwy config -- }, -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/requestRoutingRules/rr-a487ec67d2d4268de9806b8c1288c6bd", -- App Gwy config -- "name": "rr-a487ec67d2d4268de9806b8c1288c6bd", -- App Gwy config -- "properties": { -- App Gwy config -- "backendAddressPool": { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/backendAddressPools/pool-creditro-app-qa-3000-bp-3000" -- App Gwy config -- }, -- App Gwy config -- "backendHttpSettings": { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/backendHttpSettingsCollection/bp-creditro-app-qa-3000-3000-app-qa" -- App Gwy config -- }, -- App Gwy config -- "httpListener": { -- App Gwy config -- "id": "/subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/appgw-rg/providers/Microsoft.Network/applicationGateways/appgw-dev/httpListeners/fl-a487ec67d2d4268de9806b8c1288c6bd" -- App Gwy config -- }, -- App Gwy config -- "priority": 19005, -- App Gwy config -- "ruleType": "Basic" -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- ], -- App Gwy config -- "rewriteRuleSets": [], -- App Gwy config -- "sku": { -- App Gwy config -- "capacity": 1, -- App Gwy config -- "name": "Standard_v2", -- App Gwy config -- "tier": "Standard_v2" -- App Gwy config -- }, -- App Gwy config -- "sslPolicy": { -- App Gwy config -- "policyName": "AppGwSslPolicy20220101", -- App Gwy config -- "policyType": "Predefined" -- App Gwy config -- }, -- App Gwy config -- "sslProfiles": [], -- App Gwy config -- "trustedClientCertificates": [], -- App Gwy config -- "trustedRootCertificates": [], -- App Gwy config -- "urlPathMaps": [] -- App Gwy config -- }, -- App Gwy config -- "tags": { -- App Gwy config -- "aks_cluster": "subscriptions/451ee378-182f-459a-812c-c5150f0d0e42/resourceGroups/QA_cluster/providers/Microsoft.ContainerService/managedClusters/QA", -- App Gwy config -- "company": "Visma Creditro", -- App Gwy config -- "created_by": "Morten Lund Jakobsen - Devops Engineer", -- App Gwy config -- "creation": "Created at: 2025-02-07", -- App Gwy config -- "environment": "dev", -- App Gwy config -- "managed-by-k8s-ingress": "1.7.7/14189388/2025-01-15-17:06T+0530", -- App Gwy config -- "type": "Application Gateway", -- App Gwy config -- "update_at": "Created at: 2025-02-07" -- App Gwy config -- } -- App Gwy config --}
-
Morten Lund Jakobsen • 0 Reputation points
2025-02-13T10:05:38.2133333+00:00 The log is a lot bigger than what i just linked, so i dont know if what i sent is usefull :)
-
Morten Lund Jakobsen • 0 Reputation points
2025-02-14T08:07:09.03+00:00 I'm closing this request since i can't wait the response time that is included and need to finish the deployment of this gateway. Therefore I can't provided anymore logs or other things from my deployment since it has been deleted.
-
Anonymous
2025-02-14T18:21:47.13+00:00 Hi @Morten Lund Jakobsen,
I understand, and I apologize for any inconvenience caused by the response time.From the above response, I can see that the ingress is in a running state, but the IP address is not generating for the load balancer. This may be an issue on the cloud provider side, specifically regarding the number of IPs utilized in your subscription.
If you need any further assistance in the future, feel free to reach out.
Thank you.
Sign in to comment