Problems using the FHIR $import operation on resources with multiple versions

Question

Problems using the FHIR $import operation on resources with multiple versions

Peter Scott 0

I am having problems with using the FHIR $import operation on resources with multiple versions of the same resource in the import file.

Background:

I have the "initialImportMode" set to false on the FHIR Server in question.
For the resource file I am trying to import (Patient.ndjson) to test the ability to have multiple versions of a resource be imported, I have 6 different versions of the same patient in the file - this is the only data in the file.
In my Patient.ndjson file, if I provide the meta.versionId and meta.lastUpdated attributes ("meta":{"versionId":"1","lastUpdated":"1999-08-06T00:01:00.000-03:00"...), when I attempt to import the data, the import fails with the following message: "import operation failed for reason: Unable to generate internal IDs. If the lastUpdated meta element is provided as input, reduce the number of resources with the same up to millisecond lastUpdated below 10,000". This occurs even with only the versions for a single patient (6 rows in file) where the lastUpdated field is manually verified to be unique for each row.
If the meta.lastUpdated attribute is omitted, I get different behavior: the import for version #1 works, but for version #2 thru #6, the error comes back as "Failed to import duplicate resource, resource id"

Peter Scott 0 Reputation points

2025-06-09T13:05:10.4433333+00:00

Are there any plans to add support to the Azure FHIR service for $importing historical versions of resources?

Our volumes of data are large enough that an API approach to this problem of streaming in all versions of a Resource would not work giving our migration time window.
Praveen Kumar Gudipudi 1,875 Reputation points Microsoft External Staff Moderator

2025-06-10T12:21:22.8333333+00:00
Hello Peter Scott,

The incremental import mode in Azure Health Data Services' FHIR service supports ingestion of multiple versions of FHIR resources in a single batch while maintaining resource history.

Incremental mode Allows you to load resources in a nonsequential order of versions. Note for non-sequential ingestion of resources

If import files don't have the version and lastUpdated field values specified, there's no guarantee of importing all resources in the FHIR service.

If import files have resources with duplicate version and lastUpdated field values, only one resource is randomly ingested in the FHIR service.

Please refer 1.Import FHIR data.

2.https://techcommunity.microsoft.com/blog/healthcareandlifesciencesblog/incremental-mode-import-in-azure-health-data-services/3918913

Please check above suggested documents and let us know with an update.
Peter Scott 0 Reputation points

2025-06-10T12:59:32.7866667+00:00
Just to confirm - sample Patients.ndjson attached (just test data) with 6 different versions of the same patient with versionId, lastUpdated attributes - confirm no duplicate lastUpdated times - representing updates that have occurred over time with config for the incremental import mode is set as below.

"importConfiguration": { "enabled": true, "initialImportMode": false, "integrationDataStore": "sahealthdatalaketest" },
Praveen Kumar Gudipudi 1,875 Reputation points Microsoft External Staff Moderator

2025-06-13T05:23:28.6866667+00:00

Hello Peter Scott,

After configuring Patients.ndjson as you posted, are you able to see versions of resource?
Peter Scott 0 Reputation points

2025-06-13T13:00:53.4866667+00:00

Hi Praveen - no. Error message after attempting to run with the Patients.ndjson file was: "import operation failed for reason: Unable to generate internal IDs. If the lastUpdated meta element is provided as input, reduce the number of resources with the same up to millisecond lastUpdated below 10,000". Problem occurred even though only 6 versions of the same patient in the file all with different lastUpdated date.
Peter Scott 0 Reputation points

2025-06-16T16:47:01.8566667+00:00
One final update - we managed to get versions working.

InitialUpdateMode = false on the server

Make sure that any of the Ids you provide for the Patient resource are ids that have never been exchanged with the Azure FHIR Server in the past - this is what seemed to be causing the error. I assumed that running the {{fhirurl}}/:resourceType/$bulk-delete?_hardDelete=True would clean things up, however it appears this is not the case. However, as long as I generate new resource ids each time I want to $import, the system generates the patient history correctly on $import
Praveen Kumar Gudipudi 1,875 Reputation points Microsoft External Staff Moderator

2025-06-17T05:44:44.52+00:00

Hello Peter Scott,

Could you please respond to the private chat?

Here is the reference link on How to access & Data Privacy policy of private messages in Microsoft QnA.

1 answer

Your answer

Peter Scott 0 Reputation points

2025-06-09T13:05:10.4433333+00:00

Are there any plans to add support to the Azure FHIR service for $importing historical versions of resources?

Our volumes of data are large enough that an API approach to this problem of streaming in all versions of a Resource would not work giving our migration time window.
Praveen Kumar Gudipudi 1,875 Reputation points Microsoft External Staff Moderator

2025-06-10T12:21:22.8333333+00:00

Hello Peter Scott,

The incremental import mode in Azure Health Data Services' FHIR service supports ingestion of multiple versions of FHIR resources in a single batch while maintaining resource history.

Incremental mode Allows you to load resources in a nonsequential order of versions. Note for non-sequential ingestion of resources

If import files don't have the version and lastUpdated field values specified, there's no guarantee of importing all resources in the FHIR service.

If import files have resources with duplicate version and lastUpdated field values, only one resource is randomly ingested in the FHIR service.

Please refer 1.Import FHIR data.

2.https://techcommunity.microsoft.com/blog/healthcareandlifesciencesblog/incremental-mode-import-in-azure-health-data-services/3918913

Please check above suggested documents and let us know with an update.
Peter Scott 0 Reputation points

2025-06-10T12:59:32.7866667+00:00

Just to confirm - sample Patients.ndjson attached (just test data) with 6 different versions of the same patient with versionId, lastUpdated attributes - confirm no duplicate lastUpdated times - representing updates that have occurred over time with config for the incremental import mode is set as below.

"importConfiguration": { "enabled": true, "initialImportMode": false, "integrationDataStore": "sahealthdatalaketest" },
Praveen Kumar Gudipudi 1,875 Reputation points Microsoft External Staff Moderator

2025-06-13T05:23:28.6866667+00:00

Hello Peter Scott,

After configuring Patients.ndjson as you posted, are you able to see versions of resource?
Peter Scott 0 Reputation points

2025-06-13T13:00:53.4866667+00:00

Hi Praveen - no. Error message after attempting to run with the Patients.ndjson file was: "import operation failed for reason: Unable to generate internal IDs. If the lastUpdated meta element is provided as input, reduce the number of resources with the same up to millisecond lastUpdated below 10,000". Problem occurred even though only 6 versions of the same patient in the file all with different lastUpdated date.
Peter Scott 0 Reputation points

2025-06-16T16:47:01.8566667+00:00

One final update - we managed to get versions working.

InitialUpdateMode = false on the server

Make sure that any of the Ids you provide for the Patient resource are ids that have never been exchanged with the Azure FHIR Server in the past - this is what seemed to be causing the error. I assumed that running the {{fhirurl}}/:resourceType/$bulk-delete?_hardDelete=True would clean things up, however it appears this is not the case. However, as long as I generate new resource ids each time I want to $import, the system generates the patient history correctly on $import
Praveen Kumar Gudipudi 1,875 Reputation points Microsoft External Staff Moderator

2025-06-17T05:44:44.52+00:00

Hello Peter Scott,

Could you please respond to the private chat?

Here is the reference link on How to access & Data Privacy policy of private messages in Microsoft QnA.

Answer 1

Hi ,

Thanks for reaching out to Microsoft Q&A.

This is a known limitation in Azure FHIR service related to the $import operation, particularly when importing multiple versions of the same resource ID.

FHIR Versioning Support: The FHIR spec allows for multiple versions of a resource. However, many FHIR server implementations (including Azure's) do not fully support importing historical versions in one $import batch.
Azure FHIR Service $import Behavior:
- If initialImportMode = false, the server assumes data being imported may already exist and should respect existing IDs and versions.
- When you include meta.versionId and meta.lastUpdated, the server tries to persist the exact version history.
- Azure FHIR Server does not support creating multiple historical versions of the same resource via import. It only keeps the latest resource version when importing.
Specific Errors:
- Unable to generate internal IDs: This is a throttling or limitation issue triggered when too many resources have the same timestamp to the millisecond, even if resource IDs are the same.
- Failed to import duplicate resource: When you omit meta.lastUpdated, the server assumes all resources are version #1. Any duplicate resource ID is rejected.

Root Cause

The $import operation on Azure FHIR Server is designed to:

Load initial resource states into the system.

Not support importing historical versions (multiple meta.versionIds of the same resource).

Even if initialImportMode is set to false, the server:

Cannot reconstruct version history from the import file.

Always considers resource IDs to be current versions only.

Workarounds

Load only the latest version of the resource

Simply include the latest resource version per ID.

Do not try to simulate or import older versions.

Let the FHIR server auto-manage meta.versionId.

Manually replay changes via FHIR API (not via $import)

For use cases where you must preserve version history, consider a script that:

Sends each version sequentially via PUT to /Patient/{id}.

  Ensures correct order and timing to let the server assign `versionId` automatically.

Avoid setting meta.versionId in the $import

This will not give you version history but will import one version only.

All duplicates (same id) will be rejected after the first one.

Please 'Upvote'(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.

Peter Scott 0 Reputation points

2025-06-09T17:28:09.93+00:00

Reason I ask if there is any intention of supporting this in the future: the documentation seems to indicate that it's possible? Import data into the FHIR service in Azure Health Data Services | Microsoft Learn
Vinodh247 34,661 Reputation points MVP Volunteer Moderator

2025-06-10T00:55:28.6+00:00

azure's documentation implies support for historical versioning in incremental mode, the implementation today still only retains a single version per resource ID, ignoring prior versions.
Praveen Kumar Gudipudi 1,875 Reputation points Microsoft External Staff Moderator

2025-06-16T07:46:32.7766667+00:00

Hello Peter Scott,

Could you please check above comment provided by @Vinodh247 and let us know with an update.
Peter Scott 0 Reputation points

2025-06-16T10:13:42.73+00:00

Praveen - yes comment by @Vinodh247 reflects my understanding of how the system is working for us. In otherwords, the $import operation is incapable of importing multiple versions of a resource despite the documentation implying it does.

Share via

Problems using the FHIR $import operation on resources with multiple versions

1 answer

Your answer