Azure AI Search: How to Modify the document key to meet the validation requirements

Question

Azure AI Search: How to Modify the document key to meet the validation requirements

Unif 60

My Indexer has the error:

errorMessage	details
Could not parse document. Document key cannot be longer than 1024 characters.'	'Target field 'id' is either not present_ doesn't have a value set'

The Troubleshooting common indexer errors and warnings in Azure AI Search said I should Modify the document key to meet the validation requirements.

But the question is how I can modify the document key? to make document key shorter than 1024 characters?

This is my index json config:

  "fields": [
    {
      "name": "id",
      "type": "Edm.String",
      "searchable": false,
      "filterable": false,
      "retrievable": true,
      "stored": true,
      "sortable": false,
      "facetable": false,
      "key": true,
      "indexAnalyzer": null,
      "searchAnalyzer": null,
      "analyzer": null,
      "normalizer": null,
      "dimensions": null,
      "vectorSearchProfile": null,
      "vectorEncoding": null,
      "synonymMaps": []
    },
]

Laxman Reddy Revuri 5,475 Reputation points Microsoft External Staff Moderator

2024-10-15T16:23:56.0266667+00:00

Hi @Unif
Thanks for the question and using MS Q&A platform.

If the existing id field is excessively long, you can create an extra field that produces a shorter key, which may include hashing the original key or combining values from other fields.

When retrieving data from an external source, make sure the id field in your data is shortened or adjusted to be under 1024 characters.

If necessary, you can shorten your current key to meet the character limit, but be careful to avoid collisions, where two distinct records end up with the same truncated key.

explore the option of reorganizing your JSON data to utilize more concise identifiers or keys.

Ensure that your index configuration reflects any changes made to the key field.
Update your JSON index definition accordingly.
To confirm everything works smoothly:
Update Your Index Configuration: Apply this updated configuration to your Azure AI Search index.

I hope this information is helpful. If you have any further questions, please don't hesitate to reach out.
Unif 60 Reputation points

2024-10-15T23:32:13.6266667+00:00

Hi @Laxman Reddy Revuri

Thanks for answering.

I used Azure Blob Storage as an external source,

Previously, I used this method quite a long time and have no error when indexing the documents. However, when the error has come out, I quite doubt how to deal with it, because I did not change anything in Azure Blob Storage.

My files in Azure Blob Storage have simply metadata such as: user, role, etc. did not include id as metadata in Azure blob storage at all.

And this file has a Japanese name (not sure, is it also the cause?)
Laxman Reddy Revuri 5,475 Reputation points Microsoft External Staff Moderator

2024-10-16T08:58:15.21+00:00

Hi @Unif
Thanks for the question and using MS Q&A platform.
Japanese names shouldn't generally be a problem, but let’s test with simpler names to check for encoding problems.
Make sure the metadata matches the expected formats.
I hope this information is helpful.
Unif 60 Reputation points

2024-10-17T23:50:15.43+00:00

I fixed by reducing the naming of files in the virtual folder xxx/yyy/FILE_NAME/FILE_NAME--1.

The file that had a problem had a name in Non-English, but the total number of characters did not exceed the Azure blob storage limit (1024 characters).

However, when indexing it, an error occurred indicating that the key ID somehow exceeded the limit (1024 characters).

Therefore, I believe the error originated from the indexing process during encryption.

I changed the filename to xxx/yyy/FILE_NAME/1 to avoid a double filename (which was in Non-English).

As a result, the error disappeared, and the indexing process of Azure AI Search was able to proceed correctly.

Answer accepted by question author

0 additional answers

Your answer

Laxman Reddy Revuri 5,475 Reputation points Microsoft External Staff Moderator

2024-10-15T16:23:56.0266667+00:00

Hi @Unif
Thanks for the question and using MS Q&A platform.

If the existing id field is excessively long, you can create an extra field that produces a shorter key, which may include hashing the original key or combining values from other fields.

When retrieving data from an external source, make sure the id field in your data is shortened or adjusted to be under 1024 characters.

If necessary, you can shorten your current key to meet the character limit, but be careful to avoid collisions, where two distinct records end up with the same truncated key.

explore the option of reorganizing your JSON data to utilize more concise identifiers or keys.

Ensure that your index configuration reflects any changes made to the key field.
Update your JSON index definition accordingly.
To confirm everything works smoothly:
Update Your Index Configuration: Apply this updated configuration to your Azure AI Search index.

I hope this information is helpful. If you have any further questions, please don't hesitate to reach out.
Unif 60 Reputation points

2024-10-15T23:32:13.6266667+00:00

Hi @Laxman Reddy Revuri

Thanks for answering.

I used Azure Blob Storage as an external source,

Previously, I used this method quite a long time and have no error when indexing the documents. However, when the error has come out, I quite doubt how to deal with it, because I did not change anything in Azure Blob Storage.

My files in Azure Blob Storage have simply metadata such as: user, role, etc. did not include id as metadata in Azure blob storage at all.

And this file has a Japanese name (not sure, is it also the cause?)
Laxman Reddy Revuri 5,475 Reputation points Microsoft External Staff Moderator

2024-10-16T08:58:15.21+00:00

Hi @Unif
Thanks for the question and using MS Q&A platform.
Japanese names shouldn't generally be a problem, but let’s test with simpler names to check for encoding problems.
Make sure the metadata matches the expected formats.
I hope this information is helpful.
Unif 60 Reputation points

2024-10-17T23:50:15.43+00:00

I fixed by reducing the naming of files in the virtual folder xxx/yyy/FILE_NAME/FILE_NAME--1.

The file that had a problem had a name in Non-English, but the total number of characters did not exceed the Azure blob storage limit (1024 characters).

However, when indexing it, an error occurred indicating that the key ID somehow exceeded the limit (1024 characters).

Therefore, I believe the error originated from the indexing process during encryption.

I changed the filename to xxx/yyy/FILE_NAME/1 to avoid a double filename (which was in Non-English).

As a result, the error disappeared, and the indexing process of Azure AI Search was able to proceed correctly.

Answer 1

Hi @Unif,

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer.

Ask: Azure AI Search: How to Modify the document key to meet the validation requirements

Solution: I changed the filename to xxx/yyy/FILE_NAME/1 to avoid a double filename (which was in Non-English).

As a result, the error disappeared, and the indexing process of Azure AI Search was able to proceed correctly.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Share via

Azure AI Search: How to Modify the document key to meet the validation requirements

0 additional answers

Your answer