Azure AI Search Indexer + Embedding and SplitSkill not working

Question

Hello!

When I run my indexer, I get the following error:

"Skill input 'text' was '25281' tokens, which is greater then the maximum allowed '8000' tokens. Consider chunking the text with the SplitSkill in order to be able to generate embeddings for it."

So I added the SplitSkill to my skillset, but now I get this additional error:

"Optional skill input is missing or empty. Name: 'languageCode', Source: '$(/document/language)'.

Expression language parsing issues: Missing or empty value '/document/language'."

My Dataset is reading off my Sharepoint site, and this is what my Skillset JSON looks like (below):

{

  "@odata.context": "https://ch1search.search.windows.net/$metadata#skillsets/$entity",

  "@odata.etag": "\"0x8DC790E2EA903A9\"",

  "name": "skillset1715404472716",

  "description": "",

  "skills": [

    {

      "@odata.type": "#Microsoft.Skills.Text.SplitSkill",

      "name": "#2",

      "description": null,

      "context": "/document",

      "defaultLanguageCode": "en",

      "textSplitMode": "pages",

      "maximumPageLength": 1000,

      "pageOverlapLength": 0,

      "maximumPagesToTake": 0,

      "inputs": [

        {

          "name": "text",

          "source": "/document/content"

        },

        {

          "name": "languageCode",

          "source": "/document/language"

        }

      ],

      "outputs": [

        {

          "name": "textItems",

          "targetName": "pages"

        }

      ]

    },

    {

      "@odata.type": "#Microsoft.Skills.Text.AzureOpenAIEmbeddingSkill",

      "name": "EmbeddingSkill",

      "description": "Connects a deployed embedding model.",

      "context": "/document",

      "resourceUri": "https://ch1-chatbot-1025.openai.azure.com",

      "apiKey": "<redacted>",

      "deploymentId": "text-ada-002",

      "inputs": [

        {

          "name": "text",

          "source": "/document/content"

        }

      ],

      "outputs": [

        {

          "name": "embedding",

          "targetName": "vector"

        }

      ],

      "authIdentity": null

    }

  ],

  "cognitiveServices": {

    "@odata.type": "#Microsoft.Azure.Search.DefaultCognitiveServices",

    "description": null

  },

  "knowledgeStore": null,

  "indexProjections": null,

  "encryptionKey": null

}

These are the fields I have in my Index:

I'm not exactly sure how to proceed with troubleshooting. Any help would be appreciated!

Answer 1

Hi @Andrew Satanovsky looking at your Skillset JSON, it seems like you have correctly set up the SplitSkill and AzureOpenAIEmbeddingSkill. However, you might want to ensure that the source fields in your inputs arrays correctly correspond to the fields in your documents. For instance, if your documents do not have a content field, "/document/content" would not return any value, leading to errors.

Also, If your documents do not have a language field, you might need to hardcode the languageCode in your SplitSkill to ‘en’ (for English) or the appropriate language code since you this error "Optional skill input is missing or empty. Name: 'languageCode', Source: '$(/document/language)'.Expression language parsing issues: Missing or empty value '/document/language'."

When it comes to debugging/troubleshooting, Azure provides a tool called Debug Sessions that can help you identify and resolve errors in your skillset. This tool provides a visualization of your skillset and allows you to drill down to specific steps to see where an action might be failing.

Hope that helps. Please let us know if you have further questions.

-Grace