Scraping Prometheus metrics with Container insights

Question

AKS: 1.20.7
Dotnet Core: 5.0
prometheus-net: 4.2.0
prometheus-net.AspNetCore: 4.2.0

Ref: https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-prometheus-integration

I followed above link to setup to scraping of prometheus metrics. But metric doesn't show up in Azure Monitor.
Can someone please help me.

Thanks

t-shooting I did

• No errors reported in 'kubectl logs omsagent-lxdx7 -n kube-system' command
• No error reported in KubeMonAgentEvents table
• I can call my service 'http://REDACTED-load-balancer.REDACTED.svc.cluster.local:9102/metrics' without any issues

Below are the steps I did...

Relevant code snippet in startup.cs file

   public void Configure(IApplicationBuilder app,  
               IWebHostEnvironment env,  
               ILogger<Startup> logger)  
           {  
               // This is required since metrics endpoint is 9102 and rest of the service is 443  
               app.UseMetricServer(9102);  
               app.UseRouting();  
               app.UseHttpMetrics();  
               app.UseEndpoints(endpoints =>  
               {  
                   endpoints.MapControllers();  
               });  
           }  

Relevant code snippet in Deployment yaml file

   template:  
       metadata:  
         creationTimestamp: null  
         annotations:  
           prometheus.io/port: "9102"  
           prometheus.io/scrape: "true"  

Relevant change to monitoring-config.yaml file

   [prometheus_data_collection_settings.cluster]  
   fieldpass = ["timeline_messages_total", "timeline_messages_succeeded", "timeline_messages_failed", "timeline_docs_total", "timeline_docs_succeeded", "timeline_docs_failed"]  
   urls = ["http://REDACTED-load-balancer.REDACTED.svc.cluster.local:9102/metrics"]  
   kubernetes_services = [ "http://REDACTED-load-balancer.REDACTED.svc.cluster.local:9102/metrics" ]