Edit

2 - Create and load Search Index with JavaScript

  • Article
  • 5 minutes to read

Continue to build your search-enabled website by following these steps:

Create an Azure Cognitive Search resource

Create a new search resource using PowerShell and the Az.Search module. In this section, you'll also create a query key used for read-access to the index, and get the built-in admin key used for adding objects.

  1. In Visual Studio Code, open a new terminal window.

  2. Connect to Azure:

    Connect-AzAccount -TenantID <your-tenant-ID>

    Note

    You might need to provide a tenant ID, which you can find in the Azure portal in Portal settings > Directories + subscriptions.

  3. Before creating a new search service, you can list existing search services for your subscription to see if there's one you want to use:

    Get-AzResource -ResourceType Microsoft.Search/searchServices | ft

  4. Load the Az.Search module:

    Install-Module -Name Az.Search

  5. Create a new search service. Use the following cmdlet as a template, substituting valid values for the resource group, service name, tier, region, partitions, and replicas:

    New-AzSearchService -ResourceGroupName "my resource group"  -Name "myDemoSearchSvc" -Sku "Free" -Location "West US" -PartitionCount 1 -ReplicaCount 1 -HostingMode Default
    Prompt Enter
    Enter a globally unique name for the new search service. Remember this name. This resource name becomes part of your resource endpoint.
    Select a resource group for new resources Use the resource group you created for this tutorial.
    Select the SKU for your search service. Use Free for this tutorial. You can't change a SKU pricing tier after the service is created.
    Select a location for new resources. Select a region close to you.

  6. Create a query key that grants read access to a search service. Query keys have to be explicitly created. Copy the query key to Notepad so that you can paste it into the client code in a later step:

    New-AzSearchQueryKey -ResourceGroupName "my resource group"  -ServiceName "myDemoSearchSvc" -Name "mySrchQueryKey"

  7. Get the search service admin API key that was automatically created for your search service. An admin API key provides write access to the search service. Copy either one of the admin keys to Notepad so that you can use it in the bulk import step that creates and loads an index:

    Get-AzSearchAdminKeyPair  -ResourceGroupName "my resource group" -ServiceName "myDemoSearchSvc"

The script uses the Azure SDK for Cognitive Search:

  1. In Visual Studio Code, open the bulk_insert_books.js file in the subdirectory, search-website-functions-v4/bulk-insert, replace the following variables with your own values to authenticate with the Azure Search SDK:

    • YOUR-SEARCH-RESOURCE-NAME
    • YOUR-SEARCH-ADMIN-KEY
    /**
 * Add documents into Search Index
 * Script creates a new index called `good-books` in your Search resource.
 * 
 * 1. Edit the values for your own Search resource.
 *     * YOUR-SEARCH-RESOURCE-NAME
 *     * YOUR-SEARCH-ADMIN-KEY
 * 2. Run script with `npm install && npm start`
 *  
 */

const axios = require('axios');
const Papa = require('papaparse')
const { SearchClient, SearchIndexClient, AzureKeyCredential } = require("@azure/search-documents");

const SEARCH_ENDPOINT = "https://YOUR-SEARCH-RESOURCE-NAME.search.windows.net";
const SEARCH_KEY = "YOUR-SEARCH-ADMIN-KEY";

const SEARCH_INDEX_NAME = "good-books";
const SEARCH_INDEX_SCHEMA = require("./good-books-index.json");

const BOOKS_URL = "https://raw.githubusercontent.com/zygmuntz/goodbooks-10k/master/books.csv";
const BATCH_SIZE = 1000;

// Create Search service client
// used to upload docs into Index
const client = new SearchClient(
    SEARCH_ENDPOINT,
    SEARCH_INDEX_NAME,
    new AzureKeyCredential(SEARCH_KEY)
);

// Create Search service Index client
// used to create new Index
const clientIndex = new SearchIndexClient(
    SEARCH_ENDPOINT,
    new AzureKeyCredential(SEARCH_KEY)
);

// Insert docs into Search Index
// in batch
const insertData = async (data) => {

    let batch = 0;
    let batchArray = [];
    
    for (let i = 0; i < data.length; i++) {
        
        const row = data[i];
       
        // Convert string data to typed data
        // Types are defined in schema
        batchArray.push({
            "id": row.book_id,
            "goodreads_book_id": parseInt(row.goodreads_book_id),
            "best_book_id": parseInt(row.best_book_id),
            "work_id": parseInt(row.work_id),
            "books_count": !row.books_count ? 0 : parseInt(row.books_count),
            "isbn": row.isbn,
            "isbn13": row.isbn13,
            "authors": row.authors.split(",").map(name => name.trim()),
            "original_publication_year": !row.original_publication_year ? 0 : parseInt(row.original_publication_year),
            "original_title": row.original_title,
            "title": row.title,
            "language_code": row.language_code,
            "average_rating": !row.average_rating ? 0 : parseFloat(row.average_rating),
            "ratings_count": !row.ratings_count ? 0 : parseInt(row.ratings_count),
            "work_ratings_count": !row.work_ratings_count ? 0 : parseInt(row.work_ratings_count),
            "work_text_reviews_count": !row.work_text_reviews_count ? 0 : parseInt(row.work_text_reviews_count),
            "ratings_1": !row.ratings_1 ? 0 : parseInt(row.ratings_1),
            "ratings_2": !row.ratings_2 ? 0 : parseInt(row.ratings_2),
            "ratings_3": !row.ratings_3 ? 0 : parseInt(row.ratings_3),
            "ratings_4": !row.ratings_4 ? 0 : parseInt(row.ratings_4),
            "ratings_5": !row.ratings_5 ? 0 : parseInt(row.ratings_5),
            "image_url": row.image_url,
            "small_image_url": row.small_image_url
        })
        
        console.log(`${i}`);
        
        // Insert batch into Index
        if ((batchArray.length % BATCH_SIZE) === 0){
            await client.uploadDocuments(batchArray);
            
            console.log(`BATCH SENT`);
            batchArray = [];
        }
        
    }
    // Insert any final batch into Index
    if (batchArray.length > 0 ){
        await client.uploadDocuments(batchArray);
        
        console.log(`FINAL BATCH SENT`);
        batchArray = [];
    }
}
const bulkInsert = async () => {

    // Download CSV Data file
    const response = await axios.get(BOOKS_URL);
    const fileData = response.data;

    // convert CSV to JSON
    const dataObj = Papa.parse(fileData, {
        header: true,
        encoding: 'utf8',
        skipEmptyLines: true,
    })
    
    // Insert JSON into Search Index
    await insertData(dataObj.data)
}

// Create Search Index
async function createIndex() {

    SEARCH_INDEX_SCHEMA.name = SEARCH_INDEX_NAME;
    const result = await clientIndex.createIndex(SEARCH_INDEX_SCHEMA);

    console.log(result);
}

const main = async () => {

    await createIndex();
    console.log("index created");

    await bulkInsert();
}

main()
.then(() => console.log('done'))
.catch((err) => {
    console.log(`done +  failed ${err}`)
});

  2. Open an integrated terminal in Visual Studio for the project directory's subdirectory, search-website-functions-v4/bulk-insert, and run the following command to install the dependencies.

    npm install

  1. Continue using the integrated terminal in Visual Studio for the project directory's subdirectory, search-website-functions-v4/bulk-insert, to run the following bash command to run the bulk_insert_books.js script:

    npm start

  2. As the code runs, the console displays progress.

  3. When the upload is complete, the last statement printed to the console is "done".

Review the new Search Index

Once the upload completes, the search index is ready to use. Review your new index in Azure portal.

  1. In Azure portal, find the search service you created in the previous step.

  2. On the Overview page, select the Indexes tab in the middle of the page, and then select the index you created in the previous step.

    Expandable screenshot of Azure portal showing the index.

  3. By default, the index opens in the Search explorer tab. Select Search to return documents from the index.

    Expandable screenshot of Azure portal showing search results

Rollback bulk import file changes

Use the following git command in the Visual Studio Code integrated terminal at the bulk-insert directory, to roll back the changes. They aren't needed to continue the tutorial and you shouldn't save or push these secrets to your repo.

git checkout .

Copy your Search resource name

Note your Search resource name. You'll need this to connect the Azure Function app to your search resource.

Caution

While you may be tempted to use your search admin key in the Azure Function, that isn't following the principle of least privilege. The Azure Function will use the query key to conform to least privilege.

Next steps

Deploy your Static Web App