Custom Skill Set not Populating in final Index Document

Ujjwal Dalmia 0

Dear Community, need your urgent support here.

Objective

I am trying to create a custom skill in the Azure AI search. The skill is a functions all written in python. In the skill, we are using the metadata_storage_name and using this as a key (both partition and row) to extract dates from Azure Storage Table.

Issue

My functions app is generating the required skill information in the desired Output JSON - Pasting here for your reference:

{"values": [{"recordId": "a1", "data": {"from_date": "2023-07-01T00:00:00+00:00", "to_date": "9999-12-31T00:00:00+00:00"}, "errors": [], "warnings": []}]}

But this information is not getting populated in the final search index.

My Skillset definition and indexer output mapping are as follows:

Skillset Definition

{
      "@odata.type": "#Microsoft.Skills.Custom.WebApiSkill",
      "name": "gettofromdate",
      "description": "custom skill to get to date and from date for each document",
      "context": "/document",
      "uri": "https://envestdates.azurewebsites.net/api/tofromdate?code=functionskey",
      "httpMethod": "POST",
      "timeout": "PT30S",
      "batchSize": 1,
      "degreeOfParallelism": 1,
      "authResourceId": null,
      "inputs": [
        {
          "name": "pds_name",
          "source": "/document/metadata_storage_name"
        }
      ],
      "outputs": [
        {
          "name": "from_date",
          "targetName": "fromdate"
        },
        {
          "name": "to_date",
          "targetName": "todate"
        }
      ],
      "httpHeaders": {},
      "authIdentity": null
    }

Indexer Configuration - Output Field Mapping

"outputFieldMappings": [
    {
      "sourceFieldName": "/document/from_date",
      "targetFieldName": "from_date"
    },
    {
      "sourceFieldName": "/document/to_date",
      "targetFieldName": "to_date"
    }
  ],

Please suggest what am I doing wrong here?

Also, when I run a debug session, I don't get any error or warning and the skill also populates the fields

Thanks for the support

Ujjwal Dalmia

Ujjwal Dalmia 0

A small correction. The skillset definition is as follows:

{
      "@odata.type": "#Microsoft.Skills.Custom.WebApiSkill",
      "name": "#3",
      "description": "extracting from and to date",
      "context": "/document",
      "uri": "function app URI with access key",
      "httpMethod": "POST",
      "timeout": "PT30S",
      "batchSize": 1,
      "degreeOfParallelism": 1,
      "authResourceId": null,
      "inputs": [
        {
          "name": "pds_name",
          "source": "/document/metadata_storage_name"
        }
      ],
      "outputs": [
        {
          "name": "from_date",
          "targetName": "from_date"
        },
        {
          "name": "to_date",
          "targetName": "to_date"
        }
      ],
      "httpHeaders": {},
      "authIdentity": null
    }

SnehaAgrawal-MSFT 21,686

@Ujjwal Dalmia Thanks for reaching here! From what you've explained it seems everything is working. Can you confirm if the function is indeed getting invoked

Try calling the function like this:

"@odata.type": "#Microsoft.Skills.Custom.WebApiSkill",
                "description": "A custom skill that describes the image sent to OpenAI",
                "uri": "provide function URL",
                "batchSize": 4,
                "context": "/document",
                "httpHeaders": {
                    "x-functions-key": "insert key here"
                },

You can go into the Azure Function and check for invocation. Also, it helps to log things in the Azure function.

Hope this helps, let me know.

Ujjwal Dalmia 0 Reputation points

2024-07-03T19:50:47.9733333+00:00

Hi @SnehaAgrawal-MSFT ,

Thanks for your support and proactive response.

1 thing I realized is that none of the indexer runs are creating any entry in the Function invocations. I earlier thought that function invocations only get logged when you test run the code in Azure Functions.

If that's not the case, then I think the function is not getting invoked. Can you help in identifying what setting can help here?

Also, I realized that the indexer is finishing its execution in 100ms while the original run took almost 8 seconds. Can it be because there are no changes in the documents and due to this indexer is not even invoking the skillset or in simple words indexer will only work if there is incremental change? If that's the case, how can I ensure that every time there is an indexer run, all the documents are re-indexed?

Looking forward to your suggestions
SnehaAgrawal-MSFT 21,686 Reputation points

2024-07-04T06:22:05.0466667+00:00

@Ujjwal Dalmia Reached you privately.
Harini Chopperla 0 Reputation points

2024-10-04T04:23:12.76+00:00

Hi - Am facing the similar issue.

From the debug sessions, i see that the function invocation is fine and the enriched document is populating the data. However the index is not getting populated with the data.

Can someone post the solution that worked here.

Share via

Custom Skill Set not Populating in final Index Document

Your answer