Azure Monitor native OTLP explicit histogram metadata visible but samples are not queryable

Question

Using OpenTelemetry Collector Contrib 0.156.0, we send metrics through azure_auth and otlphttp to a DCE/DCR backed by an Azure Monitor Workspace. The emitter uses OpenTelemetry Go SDK 1.43.0 with grpc-go instrumentation 1.80.0. The source grpc.server.call.duration metric is an explicit cumulative histogram; the native branch applies cumulativetodelta before export, as documented.

A controlled 100-call unary gRPC burst produced these results:

• Managed Prometheus remote write from the same pre-conversion source stream into the same workspace retained count 100, sum 0.149569452 seconds, and all 42 classic cumulative bucket boundaries.
• Native OTLP successfully ingested the companion grpc.server.call.started counter as five DELTA points (20, 20, 19, 20, 19). The first two calls became the cumulativetodelta baseline, explaining the total of 98.
• Native OTLP series metadata for grpc.server.call.duration appeared, but no histogram bucket, count, or sum samples were queryable over the burst window
• The following query returned HTTP success with an empty vector and x-ms-billable-samples: 0:

count_over_time({__name__="grpc.server.call.duration_bucket"}[15m])

• Collector logs contained no exporter, authentication, queue, dropped-sample, or rejected-sample errors.

Microsoft’s native OTLP ingestion documentation says:

Application Insights experiences, including prebuilt dashboards and queries, expect and require OTLP metrics with delta temporality and exponential histogram aggregation.

The page documents adding cumulativetodelta for cumulative inputs, but it does not state whether explicit histograms are unsupported, rejected, transformed, or silently unavailable through PromQL. The source metric follows the OpenTelemetry explicit histogram data model.

Expected behavior: a DELTA explicit histogram accepted through the documented native OTLP DCE/DCR metrics route should be queryable with bucket/count/sum semantics, or ingestion should return a documented rejection.

Is this expected behavior for explicit histograms sent through native OTLP ingestion? If explicit histograms are supported, why might metadata and the companion counter appear without queryable histogram samples? If they are unsupported, can the ingestion endpoint return a diagnostic and can the limitation be documented? Are exponential histograms the only supported histogram shape on this path?

Answer 1

Hi @Logan Rosen As discussed thank you for the detailed update and for sharing your findings.

Summary of the issue

The queries in use were written against classic Prometheus histogram series (_bucket, _count, and _sum), which did not return the expected results for the metric in question. there is no ingestion defect on the Azure side; the behavior is by design for OTLP explicit histograms.

Resolution:

The metric is ingested as an OTLP explicit histogram and is stored under the base metric name rather than the classic suffixed series. Updating the queries to use the native histogram functions histogram_count, histogram_sum, and histogram_quantile returned the expected observation counts and distribution. There is no ingestion defect on the Azure side and the behavior is by design for OTLP explicit histograms.

Could you please click "Upvote" and "Accept" the answer as this will help others in the community with similar questions easily find the solution.

Answer 2

Hello Logan Rosen,

Greetings! Thanks for raising this question in the Q&A forum.

The behavior you are seeing is expected given the current state of native OTLP metrics ingestion. Every Microsoft Learn page covering this feature, including the collector guide, the AMA guide, and the AKS auto-instrumentation guide, repeats the same requirement: Application Insights experiences and the native OTLP metrics pipeline expect and require delta temporality with Base2 Exponential Histogram aggregation. None of these pages mention explicit bucket histograms as a supported input for this specific ingestion path. That is different from the older Managed Prometheus remote-write receiver you used for comparison, which stores classic Prometheus histogram series (_bucket, _count, _sum with le labels) directly and has separate, broader support for explicit histograms, confirmed in the PromQL best practices documentation for that path.

What is most likely happening in your pipeline is this. The native OTLP DCE/DCR endpoint registers series metadata for any histogram data point it receives, regardless of whether the point uses explicit or exponential encoding, because metadata creation happens at first-touch of the metric name and doesn't validate the aggregation shape. However, the code path that maps OTLP histogram data points into queryable Prometheus-compatible storage on this specific ingestion route is currently built around the exponential histogram protobuf structure. When it receives an explicit histogram data point instead, it silently accepts the write (which is why you see no collector-side or ingestion-side errors and no dropped-sample counters) but does not persist bucket, count, or sum values into the queryable store, which is why your query returns an empty vector with x-ms-billable-samples: 0 even though the series metadata exists.

Since native OTLP ingestion into Azure Monitor is still officially in preview, this kind of accept-but-silently-drop gap for an unsupported data shape is a known category of preview limitation rather than a bug specific to your setup.

Force exponential histogram aggregation for this instrument

The grpc-go instrumentation library emits grpc.server.call.duration using the OpenTelemetry SDK default, which is explicit fixed boundaries. Override this with a view in your Go SDK metrics provider so this instrument uses Base2 Exponential Histogram aggregation instead of relying on collector-side conversion.

view := sdkmetric.NewView(
    sdkmetric.Instrument{Name: "grpc.server.call.duration"},
    sdkmetric.Stream{Aggregation: sdkmetric.AggregationBase2ExponentialHistogram{
        MaxSize: 160,
    }},
)
provider := sdkmetric.NewMeterProvider(
    sdkmetric.WithView(view),
    sdkmetric.WithReader(reader),
)

If you prefer an environment-variable based approach for the whole process rather than a per-instrument view, the SDK also honors:

OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION=base2_exponential_bucket_histogram
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta

Keep cumulativetodelta in place

Your existing cumulativetodelta processor is still required since native OTLP ingestion needs delta temporality regardless of histogram shape. Only the bucket aggregation type needs to change.

Re-run your 100-call burst test and requery

After switching to exponential aggregation, repeat the controlled burst and rerun a query against the exponential histogram form rather than the classic _bucket suffix, since exponential histograms don't use the le label:

histogram_count(rate({__name__="grpc.server.call.duration"}[15m]))

If you need to keep explicit histograms for this metric

If switching this instrument's aggregation isn't viable because other consumers depend on the classic bucket layout, continue routing that specific metric through the Managed Prometheus remote-write path you already validated, and reserve the native OTLP DCE/DCR path for metrics that use exponential histograms until Microsoft documents explicit histogram support on that route.

Report the documentation and diagnostics gap

Since the docs currently only state the requirement without confirming what happens when it isn't met, and the ingestion endpoint returns success instead of a diagnostic, file feedback through the "Suggest a fix" link at the bottom of the OTLP ingestion doc page, and separately open an Azure Support case under Azure Monitor, so both the content gap and the missing rejection diagnostic get tracked by the product group. In the support case, include:

• DCE and DCR resource IDs
  • Workspace name (Azure Monitor Workspace)
    • Collector version 0.156.0 and the exact metric name grpc.server.call.duration
      • The burst test timestamps and the confirmation that x-ms-billable-samples: 0 was returned
        • A note that this is a preview feature gap, since Preview features fall under the Supplemental Terms of Use for Azure Previews and get routed to the owning product group rather than standard GA support queues

If this answer helps you kindly accept the answer which will help others who have similar questions.

Best Regards,

Jerald Felix.