Azure AI Search's SharePoint Online indexer does not index custom metadata columns, why?

Question

Azure AI Search's SharePoint Online indexer does not index custom metadata columns, why?

Manu 15

I have a document library with custom columns named "ProjectNumber" and "OwningDept". When indexed using the REST API, the index value is always NULL. What can I do to fix this?

Relevant setup info:

@Data source creation -

"container" : { "name" : "useQuery", "query" : "includeLibrary=https:./Forms/AllItems.aspx;additionalColumns=ProjectNumber,OwningDept" }

@Index creation -

{ "name": "ProjectNumber", "type": "Edm.String", "searchable": true, "filterable": false, "sortable": false, "facetable": false },
        { "name": "OwningDept", "type": "Edm.String", "searchable": true, "filterable": true, "sortable": false, "facetable": false }

@Indexer creation -

    "fieldMappings" : [
        { 
          "sourceFieldName" : "metadata_spo_site_library_item_id", 
          "targetFieldName" : "id", 
          "mappingFunction" : { 
            "name" : "base64Encode" 
          } 
        },
        { 
          "sourceFieldName" : "ProjectNumber"
        },
        { 
          "sourceFieldName" : "OwningDept"
        }
    ]

Source has ProjectNumber and OwningDept -

User's image

Result from Search Explorer

  "value": [
    {
      "@search.score": 1,
      "id": "...Z2Vvc...",
      "metadata_spo_item_name": "analysis.docx",
      "metadata_spo_item_path": "/...analysis.docx",
      "metadata_spo_item_content_type": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
      "metadata_spo_item_last_modified": "2025-03-20T15:19:49Z",
      "metadata_spo_item_size": 2527...,
      "content": "\nimage1.jpg\n\nimage10.jpg\n\nimage2.emf\n\n\n\n\n\n\n\n\n\nimage3.png\n\n",
      "ProjectNumber": null,
      "OwningDept": null
    }

Siva Nair 2,420 Reputation points Microsoft External Staff Moderator

2025-03-25T00:01:45.3166667+00:00
Hi Manu,

As you are using custom meta data column, To index custom metadata, "additionalColumns" must be specified in the query parameter of the data source.

Ensure that the field mappings in your indexer are correctly set up. It looks like you've specified the source field names "ProjectNumber" and "OwningDept", but you might need to explicitly map these to the target field names in your index.

Double-check the query used in your data source creation. Make sure that the custom columns "ProjectNumber" and "OwningDept" are included in the query and that the query is correctly formatted.

Verify that the index schema correctly defines the fields "ProjectNumber" and "OwningDept" with the appropriate data types and attributes. It looks like you've set these fields as "Edm.String", which should be fine, but ensure that the attributes like "searchable", "filterable", etc., are correctly set.

Ensure that the account used to connect to SharePoint has the necessary permissions to access the custom columns. Sometimes, permission issues can prevent the indexer from accessing certain fields.

Be aware of any known limitations of the SharePoint Online indexer. According to the documentation, there are certain limitations and considerations when using the SharePoint Online indexer, such as not supporting indexing of SharePoint Lists or .ASPX site content

https://learn.microsoft.com/en-us/azure/search/search-howto-index-sharepoint-online#include-or-exclude-by-file-type

If you have any further assistant, do let me know.
Dmitry Dyachenko 0 Reputation points

2025-03-25T09:58:01.0066667+00:00

@Siva Nair I have the same issue, 2 weeks ago everything worked fine. We have at least 2 clients with the production deployed app and the issue above is affecting them. We have not changed any configuration, it just stopped picking up values from SPO custom columns from a document library.
Manu 15 Reputation points

2025-03-25T18:09:29.3233333+00:00

Hi @Siva Nair ,

I have verified that all the points you mentioned are being followed. I have all the necessary permissions in SharePoint to be able to read data in a document library. If you have any specific permissions that I need, please let me know.

Is there a way to test the output of the datasource query?

At this point, custom column metadata is not being indexed.
Nikita Debroux 5 Reputation points

2025-03-27T16:07:55.9366667+00:00

Hello, I also have the same issue
Teresa Hoes 30 Reputation points

2025-03-27T19:38:17.41+00:00

I have the same issue. Starting March 21, no custom fields are being pulled over into the index. the values are all null. the definitions of the datasource, index, indexer have not been changed for years and has been working until that day. I have multiple sharepoint datasources/indexes and it broke the same day for all of them. Is there something going on either in Sharepoint or Azure Search that is causing this?
David Horsky 0 Reputation points

2025-03-31T13:16:07.5+00:00

Also having this issue.
I will add, that the indexer is throwing warnings about one of the custom columns being empty in some cases (I have a mapping functions connected to it), so it looks like the indexer is seeing the columns and their values as they are. It's just not writing the values into the index.
Please help, my life depends on it :D
Phu Hoang Bao 0 Reputation points

2025-04-02T11:37:09.38+00:00

Hello, I also have the same issue
Deepak Vijaykumar 0 Reputation points

2025-04-06T19:27:33.83+00:00

I've been pulling my hair for last couple of days, over the weekend trying to figure out the reason ... frustrating.

1 answer

Your answer

Siva Nair 2,420 Reputation points Microsoft External Staff Moderator

2025-03-25T00:01:45.3166667+00:00

Hi Manu,

As you are using custom meta data column, To index custom metadata, "additionalColumns" must be specified in the query parameter of the data source.

Ensure that the field mappings in your indexer are correctly set up. It looks like you've specified the source field names "ProjectNumber" and "OwningDept", but you might need to explicitly map these to the target field names in your index.

Double-check the query used in your data source creation. Make sure that the custom columns "ProjectNumber" and "OwningDept" are included in the query and that the query is correctly formatted.

Verify that the index schema correctly defines the fields "ProjectNumber" and "OwningDept" with the appropriate data types and attributes. It looks like you've set these fields as "Edm.String", which should be fine, but ensure that the attributes like "searchable", "filterable", etc., are correctly set.

Ensure that the account used to connect to SharePoint has the necessary permissions to access the custom columns. Sometimes, permission issues can prevent the indexer from accessing certain fields.

Be aware of any known limitations of the SharePoint Online indexer. According to the documentation, there are certain limitations and considerations when using the SharePoint Online indexer, such as not supporting indexing of SharePoint Lists or .ASPX site content

https://learn.microsoft.com/en-us/azure/search/search-howto-index-sharepoint-online#include-or-exclude-by-file-type

If you have any further assistant, do let me know.
Dmitry Dyachenko 0 Reputation points

2025-03-25T09:58:01.0066667+00:00

@Siva Nair I have the same issue, 2 weeks ago everything worked fine. We have at least 2 clients with the production deployed app and the issue above is affecting them. We have not changed any configuration, it just stopped picking up values from SPO custom columns from a document library.
Manu 15 Reputation points

2025-03-25T18:09:29.3233333+00:00

Hi @Siva Nair ,

I have verified that all the points you mentioned are being followed. I have all the necessary permissions in SharePoint to be able to read data in a document library. If you have any specific permissions that I need, please let me know.

Is there a way to test the output of the datasource query?

At this point, custom column metadata is not being indexed.
Nikita Debroux 5 Reputation points

2025-03-27T16:07:55.9366667+00:00

Hello, I also have the same issue
Teresa Hoes 30 Reputation points

2025-03-27T19:38:17.41+00:00

I have the same issue. Starting March 21, no custom fields are being pulled over into the index. the values are all null. the definitions of the datasource, index, indexer have not been changed for years and has been working until that day. I have multiple sharepoint datasources/indexes and it broke the same day for all of them. Is there something going on either in Sharepoint or Azure Search that is causing this?
David Horsky 0 Reputation points

2025-03-31T13:16:07.5+00:00

Also having this issue.
I will add, that the indexer is throwing warnings about one of the custom columns being empty in some cases (I have a mapping functions connected to it), so it looks like the indexer is seeing the columns and their values as they are. It's just not writing the values into the index.
Please help, my life depends on it :D
Phu Hoang Bao 0 Reputation points

2025-04-02T11:37:09.38+00:00

Hello, I also have the same issue
Deepak Vijaykumar 0 Reputation points

2025-04-06T19:27:33.83+00:00

I've been pulling my hair for last couple of days, over the weekend trying to figure out the reason ... frustrating.

Answer 1

Teresa Hoes 30

An update for anyone following this thread. I ended up putting in a ticket with MS. They applied a hotfix to our search resource and the metadata started coming over into the index again.

Juan Gonzalez 0 Reputation points

2025-04-03T18:52:53.4666667+00:00

thank you, just created a ticket myself for this, hope MS answers quickly since I pointed to this topic
Adam Horár 0 Reputation points

2025-04-04T08:18:57.15+00:00

Did they change anything in your Indexer/Index/Datasource?
Teresa Hoes 30 Reputation points

2025-04-04T14:56:03.9933333+00:00

No - nothing changed on my side. they applied a 'hotfix' to my search resource. said the root cause by the product group would be coming next week.
Adam Horár 0 Reputation points

2025-04-04T14:58:35.4666667+00:00

Thanks! :)

Share via

Azure AI Search's SharePoint Online indexer does not index custom metadata columns, why?

1 answer

Your answer