Exercise: Build and deploy an Azure AI Document Intelligence custom skill

10 minutes

In this exercise, you'll create and deploy a web service that acts as a custom skill in a AI Search pipeline. The web skill calls Azure AI Document Intelligence to extract invoice fields, such as customer names, so that they can be included in the index.

Run Cloud Shell

To start the exercise, let's connect to Cloud Shell, which you'll use to run the setup script:

In the Azure portal, select the [>_] (Cloud Shell) button at the top of the page to the right of the search box. This opens a Cloud Shell pane at the bottom of the portal.
The first time you open the Cloud Shell, you might be prompted to choose the type of shell you want to use (Bash or PowerShell). Select Bash. If you don't see this option, skip the step.
If you're prompted to create storage for your Cloud Shell, ensure your subscription is specified and select Create storage. Then wait a minute or so for the storage to be created.
Make sure the type of shell indicated on the top left of the Cloud Shell pane is switched to Bash. If it's PowerShell, switch to Bash by using the drop-down menu.
Wait for Bash to start.

Set up resources

We'll use a script to create the Azure Forms Recognizer resource, a resource group, and the AI Search resource:

In the Cloud Shell, to clone the code repository, enter this command:
```
rm -r doc-intelligence -f
git clone https://github.com/MicrosoftLearning/mslearn-ai-document-intelligence doc-intelligence
```
Tip

If you recently used this command in another lab to clone the doc-intelligence repository, you can skip this step.
Change to the 04-custom-skill directory and then execute the setup script:
```
cd doc-intelligence/Labfiles/04-custom-skill/
bash setup.sh
```

Create an Azure Function

You'll host your custom skill in an Azure Function App. Let's create that Function App:

In the Azure portal, select Create a resource.
In the Search services and marketplace textbox, type Function App and then select Function App.
In the Function App page, select Create.
In the Subscription drop-down list, select your subscription.
In the Resource group drop-down list, select FormsRecognizerResources.
In the Function app name textbox, type a unique name.
Next to Publish select Code.
In the Runtime stack drop-down list, select Python.
In the Version drop-down list, select 3.7.
In the Region drop-down list, select West US.
In the Plan type drop-down list, select Consumption (Serverless) and then select Review + create.
Review your configuration and then select Create. Azure creates the function app.

Configure the deployed Function

The Python code that you'll deploy needs to know the endpoint and API key for your Azure AI Document Intelligence resource. We'll obtain those values from the Azure portal and use the Function Apps application settings to configure them:

In the Azure portal, select All resources and then select FormsRecognizer.
Select Keys and Endpoint tab. To the right of the Endpoint textbox, select the Copy to clipboard button.
Open Notepad and paste the endpoint value.
In the Azure portal, to the right of the KEY 1 textbox, select the Copy to clipboard button.
Switch to Notepad and paste the key on a new line.
In the Azure portal, select All resources and then select the function app you created in the previous section.
Under Settings, select Configuration and then select + New application setting.
In the Add/Edit application setting dialog, in the Name textbox, type FORMS_RECOGNIZER_ENDPOINT.
Copy the endpoint value from Notepad, paste it in the Value textbox, and then select OK.
Select + New application setting.
In the Add/Edit application setting dialog, in the Name textbox, type FORMS_RECOGNIZER_KEY.
Copy the key from Notepad, paste it in the Value textbox, and then select OK.
At the top of the Configuration page, select Save and then select Continue. The function app configuration is complete.

Publish the Function

Next, deploy the Python code for the Function App:

In the Cloud Shell, change to the customskill folder:

cd ~/doc-intelligence/Labfiles/04-custom-skill/customskill

To use the Azure Functions Core Tools to deploy the Python code to your Function App, type this command and substitute <FunctionName> for the name you used when you created the Function App:
```
func azure functionapp publish <FunctionName> --python
```
When the deployment is complete, in the Azure portal, select All resources and then select your Function App.
Under Functions, select Functions. If no functions are displayed in the list, wait for a minute and then select Refresh.

Azure displays three functions: AnalyzeForm, AnalyzeInvoice, and ExtractTables.

Test the Function

We can use the Azure portal Code + Test tool to submit a test invoice to the function and see whether the inputs and output are as expected by AI Search:

In a new browser tab, browse to the sample invoice and examine the PDF file: https://github.com/MicrosoftLearning/mslearn-ai-document-intelligence/blob/main/Labfiles/04-custom-skill/SampleInvoices/Invoice_1.pdf
In the Azure portal, in the Function App, under Functions select Functions and then select AnalyzeInvoice.
Under Developer select Code + Test.
At the top of the code window, select Test/Run.
In the HTTP method drop-down list, select POST.

In the Body textbox, enter this JSON code, and then select Run:

{
    "values": [
        {
            "recordId": "record1",
            "data": { 
                "formUrl": "https://raw.githubusercontent.com/MicrosoftLearning/mslearn-ai-document-intelligence/main/Labfiles/04-custom-skill/SampleInvoices/Invoice_1.pdf",
                "formSasToken": "?st=sasTokenThatWillBeGeneratedByCognitiveSearch"
            }
        }
    ]
}

If the function is configured correctly, the output displayed includes values for CustomerAddress, CustomerAddressRecipient, and other keys. Compare the values returned with the original PDF file you opened above.

Add the Function to the AI Search skillset

To configure Azure AI Services to use the new Azure Function that calls Azure AI Document Intelligence, you must add a skillset:

In the Azure portal, select All resources and then select the Azure Function App you created in the previous section.
Next to the URL field, select the Copy to clipboard button.
Switch to Notepad and paste the URL on a new line.
In the Azure portal, under Functions, select App keys and then select Show values.
To the right of the default host key, select the Copy to clipboard button.
Switch to Notepad and paste the key on a new line.
In the Azure portal, select All resources and then select enrichedcognitivesearch.
On the Overview page for the AI Search resource, select the Skillsets tab and then select Add skillset.

Copy the following JSON code and then paste it into the Skillset definition (JSON) textbox, replacing the default content:

{ 
    "name": "[SkillsetName]",
    "description":"Skillset that calls an Azure AI Document Intelligence custom skill",
    "skills":[ 
        { 
            "@odata.type":"#Microsoft.Skills.Custom.WebApiSkill",
            "name":"formrecognizer",
            "description":"Analyze invoices and extract fields.",
            "uri":"[EndpointUrl]/api/analyze-form?code=[AzureFunctionDefaultHostKey]",
            "httpMethod":"POST",
            "timeout":"PT30S",
            "context":"/document",
            "batchSize":1,
            "inputs":[ 
                { 
                    "name":"formUrl",
                    "source":"/document/metadata_storage_path"
                },
                { 
                    "name":"formSasToken",
                    "source":"/document/metadata_storage_sas_token"
                }
            ],
            "outputs":[ 
                { 
                    "name":"address",
                    "targetName":"address"
                },
                { 
                    "name":"recipient",
                    "targetName":"recipient"
                }
            ]
        }
    ],
    "cognitiveServices": {
        "@odata.type": "#Microsoft.Azure.Search.DefaultCognitiveServices"
    }
}

Replace [SkillsetName] with a unique name for your skillset.
From Notepad, copy the Function App URL and paste it into the JSON code, replacing [EndpointUrl].
From Notepad, copy the default host key and paste it into the JSON code, replacing [AzureFunctionDefaultHostKey].
At the top left, select Save. You've completed the integration of AI Search with Forms Recognizer.

Clean up

Let's remove the exercise resources from your Azure subscription:

In Azure portal, select Resource groups.
Select FormsRecognizerResources and then select Delete resource group.
In the TYPE RESOURCE GROUP NAME textbox, type FormsRecognizerResources and then select Delete. Azure removes the resource group, the Azure AI Document Intelligence resource, the AI Search resource, and their associated resources.