Azure IoT Hub device queries returns stale data

Question

Azure IoT Hub device queries returns stale data

Stian Sandve 20

We are using Azure IoT Hub and have integrations running on dotnet using the C# SDK. Lately we have been experiencing issues where the GetNextAsTwinAsync returns old versions of the device twin for some devices. It is not a matter of eventual consistency, because the returned twins are several days old. When I fetch the twin directly using GetTwinAsync, I get the most recent twin.

I have isolated it to the Azure IoT Hub API, because I experience the same from the Azure Portal or if I use the /query API using Postman.

Example:

POST https://our-iot-hub.net/devices/query?api-version=2024-03-31

BODY

{"query":"SELECT * FROM devices WHERE deviceId = 'FOOBAR'"}

RESPONSE

[
	{
		"deviceId": "FOOBAR",
		"etag": "AAAAAAAAAHU=",
		"deviceEtag": "NTI5MjY0NzMy",
		"status": "enabled",
		"statusUpdateTime": "0001-01-01T00:00:00Z",
		"connectionState": "Connected",
		"lastActivityTime": "2025-11-12T19:33:32.2807186Z",
		"cloudToDeviceMessageCount": 0,
		"authenticationType": "sas",
		"x509Thumbprint": {
			"primaryThumbprint": null,
			"secondaryThumbprint": null
		},
		"modelId": "",
		"version": 5031,
...

When I fetch the twin directly by device ID, I get the most recent version:

GET https://out-iot-hub.net/twins/FOOBAR?api-version=2024-03-31

RESPONSE

{
	"etag": "AAAAAAAAAMA=",
	"deviceId": "FOOBAR",
	"deviceEtag": "NTI5MjY0NzQw",
    "status": "enabled",
	"statusUpdateTime": "0001-01-01T00:00:00.0000000Z",
	"lastActivityTime": "2025-11-17T10:00:02.1081882Z",
	"connectionState": "Connected",
	"version": 5335,
...

I am seeing the same results with api version 2021-04-12, which is what the SDK uses.This is a major issue for us, and causes all kinds of bugs related to outdated desired/reported properties. Just to be clear, we are not having issues with direct twin reads and writes. It works as expected. The problem is that device queries operate on very old versions of the device twin for some of our devices.

Answer accepted by question author

1 additional answer

Your answer

Answer 1

Hello Stian Sandve,

Welcome to Microsoft Q&A and Thank you for reaching out.

I understand that you’re experiencing an issue in Azure IoT Hub where GetNextAsTwinAsync (query-based twin retrieval) is returning outdated device twins, while GetTwinAsync (direct twin read) always returns the latest state. This mismatch can understandably create bugs and inconsistent behavior in your application.

Based on your testing through the SDK, REST API, and Azure Portal, this behavior is not expected. Device twin queries should return near-real-time data not versions that are several days old.

This issue occurs due to a known limitation in how IoT Hub handles device queries versus direct reads. Direct twin reads (GetTwinAsync) always fetch the latest live state from the service. However, query operations (/devices/query or GetNextAsTwinAsync) rely on an internal indexed store that is updated asynchronously. Normally, this index refreshes within seconds or minutes, but in your case the index appears stale for days, which is abnormal and indicates a backend issue.

The root cause is typically tied to indexing delays or failures within IoT Hub. The internal index may stop updating for some devices if the service is experiencing throttling, high ingestion rates, or a stuck update pipeline. In many cases, this issue occurs at the partition level IoT Hub stores devices across partitions, and if one partition's indexer fails, only those devices return stale data via queries.

Changing API versions (e.g., 2021-04-12 vs 2024-03-31) won’t resolve the issue because the underlying query-indexing mechanism is the same across all API versions.

To diagnose the issue, comparing the version from query results with direct reads as you’ve done is the correct first step.

If the difference is large (multiple days), this is not standard eventual consistency and clearly points to an indexing problem. You can also review IoT Hub metrics in Azure Monitor, such as TwinUpdateOperations, QueryOperations, throttling events, and errors.

Running queries for multiple devices that belong to the same partition can also confirm if the problem is isolated to a specific partition.

As a temporary workaround, avoid relying on queries for real-time or fast-changing twin data. Instead, use GetTwinAsync wherever accurate and latest state is required.

If you need to perform batch or large-scale processing, consider using device twin change notifications (Event Grid or routes) to maintain your own up-to-date cache, and reserve queries for static or slowly changing metadata such as tags or deviceId lists.

Please refer this

IoT Hub query language for device and module twins

Issues with twin reads or updates

Common error codes

I Hope this helps. Do let me know if you have any further queries.

Thank you!

Answer 2

Stian Sandve 20

First of all, thanks for the fast response! I was not expecting you to reply that fast, so I already started on a workaround. I actually went for what you suggested and added a route for device twin updates that sends messages to a service bus topic that I subscribe to. I use this, in combination with a period full sync to keep a copy of all device twins in Cosmos DB. I think this will work well, but it sure did cost me valuable time to implement this workaround. It also adds operational / infrastructure cost.

I browsed the IoT Hub logs and found no evidence of throttling. I also tried to upgrade from S1 to S2 to see if that would trigger a new indexing process, but the index was still broken. There is no way for me to manually "restart" or "clear cache" on the IoT Hub service, which I understand. It does however feel like your hands are tied when problems like this arise on a managed service. You definitely loose faith, which is why I had to go with the separate database solution...

SRILAKSHMI C 10,640 Reputation points Microsoft External Staff Moderator

2025-11-18T08:29:28.53+00:00
Hi Stian Sandve,

Thank you for the detailed follow-up, and I completely understand your frustration here. You’ve already put significant time and effort into implementing a workaround, and it’s absolutely valid to expect the managed service to keep the device twin index consistent without requiring additional infrastructure on your side.

You’ve done all the correct diagnostic steps:

Checking IoT Hub logs for throttling

Verifying no ingestion pressure

Testing with different API versions

Even scaling from S1 → S2 to trigger backend refresh

The fact that the query index still returns stale twins after several days strongly indicates an internal indexing issue on the IoT Hub backend, not something in your application or configuration. This is not expected behavior, and it cannot be corrected from your side (there’s no customer-available cache purge, index rebuild, or partition refresh option).

Because the issue persists across:

The SDK (GetNextAsTwinAsync)

REST query API (/devices/query)

Azure Portal query UI

…it confirms that the problem is occurring inside IoT Hub’s query index pipeline.

At this point, the right next step is for us to escalate this to the IoT Hub engineering team so they can:

Inspect the affected hub’s partitions

Verify index health status

Identify any stuck or lagging index workers

Perform the necessary backend refresh or rebuild

Prevent this from recurring

Your workaround using twin change notifications + Cosmos DB is a solid pattern, and it will help you stay resilient, but you should not have needed to implement this specifically due to stale data. I agree that it adds operational overhead that shouldn’t be required in normal circumstances.

Thank you!
Stian Sandve 20 Reputation points

2025-11-18T09:09:01.8+00:00
Thanks again for confirming the behaviour. I’m almost done migrating away from functions that rely on the /devices/query endpoint. I have one more concern regarding how Jobs evaluate devices.

We use Jobs to schedule twin updates through:

await jobClient.ScheduleTwinUpdateAsync(jobId, queryCondition, updatedTwin, new DateTime(jobStartTime.Ticks, DateTimeKind.Utc), (int)TimeSpan.FromHours(1).TotalSeconds, cancellationToken);

As you can see, we pass a query condition when scheduling the job.

My question is:

Does the query inside ScheduleTwinUpdateAsync use the same eventually-consistent device twin index as /devices/query and GetNextAsTwinAsync?

If so, can Job queries also suffer from the same indexing issues that cause stale or outdated device twins to appear during selection?
SRILAKSHMI C 10,640 Reputation points Microsoft External Staff Moderator

2025-11-19T14:02:27.2166667+00:00
Hi Stian Sandve,

Thank you for raising this follow-up.

Yes, IoT Hub Jobs also rely on the same eventually-consistent device twin query index that is used by:

/devices/query

GetNextAsTwinAsync

Azure Portal twin queries

This means Jobs can be affected by the same stale-index behavior when selecting devices based on a query condition.

When you schedule a Job using:

await jobClient.ScheduleTwinUpdateAsync(jobId, queryCondition, updatedTwin, ...);

IoT Hub evaluates the queryCondition against the same internal indexed projection of device twins. This index is normally updated very frequently, but as you’ve observed when the index becomes stale for a partition, Jobs may evaluate against outdated twin values as well.

This affects only device selection for the job, not the update operation itself. In other words:

The job will correctly apply the update to the devices it selects

But the set of selected devices may be incorrect if the index is stale (devices missing or devices included incorrectly)

Since your hub is currently showing significantly stale index data, Job queries may not behave reliably if they depend on rapidly changing desired/reported properties.

If your Job queries rely on static metadata like tags or deviceId patterns, the risk is low. If they rely on dynamic twin properties, the selection may be inaccurate until the backend index issue is fixed.

While the engineering team investigates the backend indexing issue, I recommend one of the following patterns:

Use your Cosmos DB twin cache to determine the device set, then run the Job with an explicit device list instead of a query condition.

If you need to continue with Jobs + query conditions, limit query filters to stable metadata (tags or deviceId filters), not dynamic reported/desired properties.

Thank you!
Stian Sandve 20 Reputation points

2025-11-20T13:37:26.03+00:00

OK. Thank you for the clarification and recommendations.

All twins are now up to date in the query index again. I’ve adjusted my system so I don’t rely on query-based operations for real-time state anymore. If the index becomes stale again, I will also switch away from Jobs that depend on query conditions.
SRILAKSHMI C 10,640 Reputation points Microsoft External Staff Moderator

2025-11-21T10:39:54.62+00:00

Hi Stian Sandve,

Thanks for the update, and I’m really glad to hear that the twins in the query index are now up to date again.

It’s great that you’ve already adjusted your system to avoid relying on query-based operations for real-time state, and your plan to fall back from Job queries if the index becomes stale again is solid. Those patterns will help ensure your solution stays resilient even if the IoT Hub query index experiences delays in the future.

That said, this behavior is not expected, and your experience has been fully valid.

Thanks again for your patience and for working through this in such a systematic way.I converted my last comment to answer. Could you please take a moment to retake the survey by accepting the below response? Your feedback is greatly appreciated.

Thank you!

Share via

Azure IoT Hub device queries returns stale data

1 additional answer

Your answer