Mesh 201 Tutorial Chapter 6: Get answers to questions using Azure OpenAI

Article
09/25/2024

In this chapter, we move on to the fifth and final station where you'll learn how to implement an Azure OpenAI-based virtual assistant or "chatbot". Note that you could use any AI service you want here (for example, Copilot Studio). Keeping with the theme of searching for a location for a wind farm, the AI assistant will be customized to answer wind farm-related questions.

Important notes:

OpenAI is only available to approved enterprise customers and partners. If you're not yet in this group, you must submit a registration form. Learn more about the registration process.
If you run into issues with subscriptions and OpenAI, you may find an answer by posting in our Mesh Creator Forum.

Setting up for this station

In order to complete this station, you'll need to visit the Azure Portal and then get a URI (also called "Endpoint") and a key for your tenant. Later, you'll insert the key into some code that'll enable you to call Azure OpenAI. Let's do this step now so that you don't have to interrupt your workflow later on.

Create and deploy an Azure OpenAI resource

In your browser, navigate to the Azure Portal and then log in.
In the Search box at the top of the window, type in "azure openai" and then press the Enter key. This takes you to the Azure AI services | Azure OpenAI page.
Click the Create button.

IMPORTANT: For many of the settings detailed below, we don't make a specific recommendation. You should choose the options that make the most sense for you and your organization.
On the Basics page, create names where requested, and choose the options you want for Subscription, Region, and Pricing tier.
On the Network page, choose the option you want.
On the Tags page, you can add tags or skip these options.
On the Review and Submit page, review the information and then click the Create button.

The resource deploys and you should see a message saying that the deployment is in progress. Eventually, you'll see a Deployment is complete page.

__________________________________

Deploy the model in Azure OpenAI Studio

Click the Go to Resource button.
On the Resource page, click Go to Azure OpenAI Studio.
In the left-side menu under Management, select Deployments.
Select Create new deployment.
Click the Deploy Model drop-down and then select Deploy base Model.
In the Select a model dialog, select "gpt-35-turbo".
If you wish, take a moment to view the description. When you're finished, click the Confirm button.

Note: You can choose a different model, but this would require some changes to the code.

In the Deployment name field, type in "gpt-35-turbo".
For the other settings, choose the names and options that make the most sense for you and your organization. When you're finished, click the Deploy button.

Copy the URI and key

Navigate back to the Azure main page, and then under Resources, select the resource you created.
On the Resource page in the left-side menu, select Keys and Endpoint.
On the Keys and Endpoint page, click the "Copy to clipboard" button for KEY 1 or KEY 2 (it doesn't matter which one) and then paste it into a text file using Windows Notepad or another text editor.
Click the "Copy to clipboard" button for Endpoint and paste it into the same text file.
Save the text file. You'll need these two pieces of information later in the tutorial.

Add the prefab for Station 5

In the Project folder, navigate to Assets > MeshCloudScripting and then drag the 5 - AIAssistant prefab to the Hierarchy and place it as a child object to Mesh Cloud Scripting and just under 4 - GlobeWithCloudScripting.

Insert the URI and API information for Azure OpenAI

In the Hierarchy, select the Mesh Cloud Scripting GameObject.
In the Inspector, navigate to the Mesh Cloud Scripting component and then click Open application folder. This opens the project folder that contains the files for Mesh Cloud Scripting in the Windows File Explorer.
Open the file named appsettings.UnityLocalDev.json in your code editor. The last two lines of code contain placeholder comments for Azure OpenAI configuration settings.
Paste the URI and key that you copied earlier from the Azure Portal into these two lines, replacing the placeholder comments.
Save and close the file.

Update the Directory.packages.props file

In the File Explorer window that displays the Mesh Cloud Scripting files, open the file named Directory.Packages.props in your code editor. Note the comment about adding package references.

Replace the comment with the line below:

<PackageVersion Include="Azure.AI.OpenAI" Version="1.0.0-beta.15"/>

Save and close the file.

Update the csproj file

In the File Explorer window that displays the Mesh Cloud Scripting files, open the file named StartingPoint.csproj in your code editor.
Note that at the bottom of the file, just above the ItemGroup element with the WeatherAPI information, there's a comment with a placeholder for a package reference.
Delete the comment and replace it with the line below:
```
<PackageReference Include="Azure.AI.OpenAI" />
```
Save and close the file.

Add the code that enables OpenAI

In the File Explorer window that displays the Mesh Cloud Scripting files, navigate to the StartingPoint folder and then open the file named App.cs in your code editor.
In the App.cs file, find the "Paste code here" comment at the top of the list of using directives.
Copy the code below.
```
using Azure;
using Azure.AI.OpenAI;
```
Replace the "Paste code here" comment you just found with the code you copied.
Find the "Paste code here" comment located below the _weatherAPIKey_ field.
Copy the code below.
```
private OpenAIClient _openAIClient;
```
Replace the "Paste code here" comment you just found with the code you copied.
Find the "Paste code here" comment located in the body of the constructor.

Copy the code below.

Uri azureOpenAIResourceUri = new(configuration.GetValue<string>("AZURE_OPENAI_API_URI"));
AzureKeyCredential azureOpenAIApiKey = new(configuration.GetValue<string>("AZURE_OPENAI_API_KEY"));
_openAIClient = new(azureOpenAIResourceUri, azureOpenAIApiKey);

Replace the "Paste code here" comment you just found with the code you copied.
Find the "Paste code here" comment that follows the refreshButtonNode if statement inside the StartAsync() method.

Copy the code below.

var aiParentNode = _app.Scene.FindFirstChild("5 - AIAssistant", true) as TransformNode;
var infoButton = aiParentNode?.FindFirstChild<InteractableNode>(true);

if (infoButton != null)
{
    infoButton.Selected += async (sender, args) =>
    {
        // Ensure we have weather data before beginning the conversation
        await GetCurrentWeather(_latlong);

        // Display an input dialog for the user to send a message to the large language model (LLM)
        // Paste code here
    };
}

Replace the "Paste code here" comment you just found with the code you copied.

Add the code that displays the OpenAI input dialog

Find the "Paste code here" comment located shortly after the GetCurrentWeather() method in the infoButton if statement.

Copy the code below.

await _app.ShowInputDialogToParticipantAsync("Ask Azure OpenAI", args.Participant).ContinueWith(async (response) =>
{
    try
    {
        if (response.Exception != null)
        {
            throw response.Exception.InnerException ?? response.Exception;
        }

        string participantInput = response.Result;

        // Paste code here

        // Wait for a response from OpenAI based on the user's message
        // Paste code here
    }
    catch (Exception ex)
    {
        var log = $"Exception during OpenAI request: {ex.Message}";
        _logger.LogCritical(log);

        if (!response.IsCanceled)
        {
            _app.ShowMessageToParticipant(log, args.Participant);
        }
    }
}, TaskScheduler.Default);

Replace the "Paste code here" comment you just found with the code you copied.

The code does the following:
- Call Mesh Cloud Scripting through the method named ShowInputDialogToParticipantAsync(). The arguments are the message you want to prompt the user with ("Ask Azure OpenAI") and who you want to display the input dialog to (args.Participant).
- When the input dialog pops up, check for errors.
- Save whatever the participant typed in (the result) to participantInput as a String.

Send GPT-3.5 Turbo the result of the input dialog

The code below sends the GPT-3.5 Turbo model the result of the input dialog with instructions on how to respond with the current weather data.

Find the first "Paste code here" comment in the try...catch block you just pasted.

Copy the code below:

    var chatCompletionsOptions = new ChatCompletionsOptions()
    {
        DeploymentName = "gpt-35-turbo", // Use DeploymentName for "model" with non-Azure clients
        Messages =
        {
            // The system message represents instructions or other guidance about how the assistant should behave
            new ChatRequestSystemMessage(
                "You are a helpful assistant." +
                "You're part of a developer sample for the Mesh Toolkit." +
                "Use brief answers, less than 1 paragraph." +
                "You can suggest a good location for a wind farm based on current and historical weather data." +
                "We're looking at globe with the current weather data displayed for each of these locations:  Lagos Nigeria, Redmond WA USA, Dublin Ireland" +
                "Current weather conditions for these locations:" + _currentWeatherText
                ),
            new ChatRequestUserMessage(participantInput),
        }
    };

Replace the "Paste code here" comment you just found with the code you copied.

The code does the following:
- Create an instance of the ChatCompletionOptions class named chatCompletionOptions.
- Initialize a variable named DeploymentName with the deployment name you created earlier ("gpt-35-turbo").
- Initialize a new instance of ChatRequestSystemMessage with guidance on how the assistant should answer questions. This includes the locations with the weather data you're interested in and the current conditions (_currentWeatherText) for those locations. The value of _currentWeatherText was received when we called weatherapi.com in Chapter 5.
- Initialize a new instance of ChatRequestUserMessage with the question the participant asked.
- Send the information about the request to the LLM (Large Language Model).

Add the code that displays the response from the LLM

Find the remaining "Paste code here" comment in the try...catch block you just pasted.

Copy the code below:

        var aiResponse = await _openAIClient.GetChatCompletionsAsync(chatCompletionsOptions);

        // Display the first response from the LLM
        var responseMessage = aiResponse.Value.Choices[0].Message;
        _app.ShowMessageToParticipant($"<i>You asked: {participantInput}</i>\n\nResponse: {responseMessage.Content}", args.Participant);

Replace the "Paste code here" comment you just found with the code you copied.

The code does the following:
- Wait for the response from the LLM.
- The LLM sends back several responses in an array (responseMessage). You can choose the one you want to show.
- Call ShowMessageToParticipant() in the Mesh Cloud Scripting API to display the response.
Save and close the file.

Test your work

In the Unity Editor, save the project and then press the Play button.
Your avatar is spawned on the side of the Sphere Terrace that contains the first three stations. Navigate to the opposite side of the Sphere Terrace and then position yourself in front of Station 5.
Click the Info button located in the information text box for Station 5.
When the Ask Azure OpenAI dialog appears, type in a question.
The response appears in the dialog. When you're finished, click OK.

Conclusion

Congratulations! In this Mesh 201 tutorial, you learned about the following:

Loading local shared and non-shared HTML files into a WebSlate.
Using a 3D asset to call a Web API and download data into a WebSlate.
Pull data from internal or public sources into your scene and display it in 3D.
Set up AI-powered interactions using an Azure OpenAI-based virtual assistant or "chatbot".

Now you can put your new Mesh skills to work and build collaborative experiences that are even more useful and exciting!

Next Steps

Build and Publish

If you want to build and publish this tutorial project, do the following:

Go to our article on preparing a Mesh Cloud Scripting project and then follow the instructions to record your Resource Group and Subscription ID.
Go to our article on building and publishing your environment and then follow the instructions there. At a certain point, you'll have to navigate to a supplementary article with specialized instructions for building a project with Mesh Cloud Scripting and then return to the main build and publish article. This flow is all laid out for you in the articles.
If you want to, you can test your environment in the Mesh app.

Learn more about WebSlates

Visit the WebSlates article on the Mesh Help site.

Read a blog post written by the WebSlates product manager.

Challenge

Try creating your own station with a WebSlate and a button that loads an HTML file. Be sure to share your efforts in our Mesh Developer Forum!

Share via

Mesh 201 Tutorial Chapter 6: Get answers to questions using Azure OpenAI

Setting up for this station

Create and deploy an Azure OpenAI resource

Deploy the model in Azure OpenAI Studio

Copy the URI and key

Add the prefab for Station 5

Insert the URI and API information for Azure OpenAI

Update the Directory.packages.props file

Update the csproj file

Add the code that enables OpenAI

Add the code that displays the OpenAI input dialog

Send GPT-3.5 Turbo the result of the input dialog

Add the code that displays the response from the LLM

Test your work

Conclusion

Next Steps

Feedback

Additional resources