Edit

Map enriched output to fields in a search index in Azure AI Search

  • Article

Indexer Stages

This article explains how to set up output field mappings, defining a data path between in-memory data structures created during skillset processing, and target fields in a search index. An output field mapping is defined in an indexer and has the following elements:

"outputFieldMappings": [
  {
    "sourceFieldName": "document/path-to-a-node-in-an-enriched-document",
    "targetFieldName": "some-search-field-in-an-index",
    "mappingFunction": null
  }
],

In contrast with a fieldMappings definition that maps a path between two physical data structures, an outputFieldMappings definition maps in-memory enrichments to fields in a search index.

Output field mappings are required if your indexer has an attached skillset that creates new information, such as text translation or key phrase extraction. During indexer execution, AI-generated information exists in memory only. To persist this information in a search index, you'll need to tell the indexer where to send the data.

Output field mappings can also be used to retrieve specific nodes in a source document's complex type. For example, you might want just "FullName/LastName" in a multi-part "FullName" property. When you don't need the full complex structure, you can flatten individual nodes in a nested data structures, and then use an output field mapping to send the output to a string collection in your search index.

Output field mappings apply to:

  • In-memory content that's created by skills or extracted by an indexer. The source field is a node in an enriched document tree.

  • Search indexes. If you're populating a knowledge store, use projections for data path configuration. If you're populating vector fields, output field mappings aren't used.

Output field mappings are applied after skillset execution or after document cracking if there's no associated skillset.

Define an output field mapping

Output field mappings are added to the outputFieldMappings array in an indexer definition, typically placed after the fieldMappings array. An output field mapping consists of three parts.

"fieldMappings": []
"outputFieldMappings": [
  {
    "sourceFieldName": "/document/path-to-a-node-in-an-enriched-document",
    "targetFieldName": "some-search-field-in-an-index",
    "mappingFunction": null
  }
],
Property Description
sourceFieldName Required. Specifies a path to enriched content. An example might be /document/content. See Reference enrichments in an Azure AI Search skillset for path syntax and examples.
targetFieldName Optional. Specifies the search field that receives the enriched content. Target fields must be top-level simple fields or collections. It can't be a path to a subfield in a complex type. If you want to retrieve specific nodes in a complex structure, you can flatten individual nodes in memory, and then send the output to a string collection in your index.
mappingFunction Optional. Adds extra processing provided by mapping functions supported by indexers. For enrichment nodes, encoding and decoding are the most commonly used functions.

You can use the REST API or an Azure SDK to define output field mappings.

Tip

Indexers created by the Import data wizard include output field mappings generated by the wizard. If you need examples, run the wizard over your data source to see the rendered definition.

Use Create Indexer (REST) or Update Indexer (REST), any API version.

This example adds entities and sentiment labels extracted from a blob's content property to fields in a search index.

PUT https://[service name].search.windows.net/indexers/myindexer?api-version=[api-version]
Content-Type: application/json
api-key: [admin key]
{
    "name": "myIndexer",
    "dataSourceName": "myDataSource",
    "targetIndexName": "myIndex",
    "skillsetName": "myFirstSkillSet",
    "fieldMappings": [],
    "outputFieldMappings": [
        {
            "sourceFieldName": "/document/content/organizations/*/description",
            "targetFieldName": "descriptions",
            "mappingFunction": {
                "name": "base64Decode"
            }
        },
        {
            "sourceFieldName": "/document/content/organizations",
            "targetFieldName": "orgNames"
        },
        {
            "sourceFieldName": "/document/content/sentiment",
            "targetFieldName": "sentiment"
        }
    ]
}

For each output field mapping, set the location of the data in the enriched document tree (sourceFieldName), and the name of the field as referenced in the index (targetFieldName). Assign any mapping functions needed to transform the content of a field before it's stored in the index.

Flatten complex structures into a string collection

If your source data is composed of nested or hierarchical JSON, you can't use field mappings to set up the data paths. Instead, your search index must mirror the source data structure for at each level for a full import.

This section walks you through an import process that produces a one-to-one reflection of a complex document on both the source and target sides. Next, it uses the same source document to illustrate the retrieval and flattening of individual nodes into string collections.

Here's an example of a document in Azure Cosmos DB with nested JSON:

{
   "palette":"primary colors",
   "colors":[
      {
         "name":"blue",
         "medium":[
            "acrylic",
            "oil",
            "pastel"
         ]
      },
      {
         "name":"red",
         "medium":[
            "acrylic",
            "pastel",
            "watercolor"
         ]
      },
      {
         "name":"yellow",
         "medium":[
            "acrylic",
            "watercolor"
         ]
      }
   ]
}

If you wanted to fully index the above source document, you'd create an index definition where the field names, levels, and types are reflected as a complex type. Because field mappings aren't supported for complex types in the search index, your index definition must mirror the source document.

{
  "name": "my-test-index",
  "defaultScoringProfile": "",
  "fields": [
    { "name": "id", "type": "Edm.String", "searchable": false, "retrievable": true, "key": true},
    { "name": "palette", "type": "Edm.String", "searchable": true, "retrievable": true },
    { "name": "colors", "type": "Collection(Edm.ComplexType)",
      "fields": [
        {
          "name": "name",
          "type": "Edm.String",
          "searchable": true,
          "retrievable": true
        },
        {
          "name": "medium",
          "type": "Collection(Edm.String)",
          "searchable": true,
          "retrievable": true,
        }
      ]
    }
  ]
}

Here's a sample indexer definition that executes the import (notice there are no field mappings and no skillset).

{
  "name": "my-test-indexer",
  "dataSourceName": "my-test-ds",
  "skillsetName": null,
  "targetIndexName": "my-test-index",

  "fieldMappings": [],
  "outputFieldMappings": []
}

The result is the following sample search document, similar to the original in Azure Cosmos DB.

{
  "value": [
    {
      "@search.score": 1,
      "id": "240a98f5-90c9-406b-a8c8-f50ff86f116c",
      "palette": "primary colors",
      "colors": [
        {
          "name": "blue",
          "medium": [
            "acrylic",
            "oil",
            "pastel"
          ]
        },
        {
          "name": "red",
          "medium": [
            "acrylic",
            "pastel",
            "watercolor"
          ]
        },
        {
          "name": "yellow",
          "medium": [
            "acrylic",
            "watercolor"
          ]
        }
      ]
    }
  ]
}

An alternative rendering in a search index is to flatten individual nodes in the source's nested structure into a string collection in a search index.

To accomplish this task, you'll need an outputFieldMappings that maps an in-memory node to a string collection in the index. Although output field mappings primarily apply to skill outputs, you can also use them to address nodes after "document cracking" where the indexer opens a source document and reads it into memory.

Below is a sample index definition, using string collections to receive flattened output:

{
  "name": "my-new-flattened-index",
  "defaultScoringProfile": "",
  "fields": [
    { "name": "id", "type": "Edm.String", "searchable": false, "retrievable": true, "key": true },
    { "name": "palette", "type": "Edm.String", "searchable": true, "retrievable": true },
    { "name": "color_names", "type": "Collection(Edm.String)", "searchable": true, "retrievable": true },
    { "name": "color_mediums", "type": "Collection(Edm.String)", "searchable": true, "retrievable": true}
  ]
}

Here's the sample indexer definition, using outputFieldMappings to associate the nested JSON with the string collection fields. Notice that the source field uses the path syntax for enrichment nodes, even though there's no skillset. Enriched documents are created in the system during document cracking, which means you can access nodes in each document tree as long as those nodes exist when the document is cracked.

{
  "name": "my-test-indexer",
  "dataSourceName": "my-test-ds",
  "skillsetName": null,
  "targetIndexName": "my-new-flattened-index",
  "parameters": {  },
  "fieldMappings": [   ],
  "outputFieldMappings": [
    {
       "sourceFieldName": "/document/colors/*/name",
       "targetFieldName": "color_names"
    },
    {
       "sourceFieldName": "/document/colors/*/medium",
       "targetFieldName": "color_mediums"
    }
  ]
}

Results from the above definition are as follows. Simplifying the structure loses context in this case. There's no longer any associations between a given color and the mediums it's available in. However, depending on your scenario, a result similar to the one shown below might be exactly what you need.

{
  "value": [
    {
      "@search.score": 1,
      "id": "240a98f5-90c9-406b-a8c8-f50ff86f116c",
      "palette": "primary colors",
      "color_names": [
        "blue",
        "red",
        "yellow"
      ],
      "color_mediums": [
        "[\"acrylic\",\"oil\",\"pastel\"]",
        "[\"acrylic\",\"pastel\",\"watercolor\"]",
        "[\"acrylic\",\"watercolor\"]"
      ]
    }
  ]
}

See also