Create Data Source (Azure AI Search REST API)

Article
10/06/2023

Important

This API reference is for a legacy version. See Data plane REST operations for updated reference documentation. Use the filter on the top left to select a version.

In Azure AI Search, a data source is used with indexers, providing the connection information for on demand or scheduled data refresh of a target index, pulling data from supported Azure data sources.

You can use either POST or PUT on the request. For either one, the JSON document in the request body provides the object definition.

POST https://[service name].search.windows.net/datasources?api-version=[api-version]  
    Content-Type: application/json  
    api-key: [admin key]

Alternatively, you can use PUT and specify the name on the URI.

PUT https://[service name].search.windows.net/datasources/[data source name]?api-version=[api-version]
    Content-Type: application/json  
    api-key: [admin key]

HTTPS is required for all service requests. If the object doesn't exist, it's created. If it already exists, it's updated to the new definition.

Note

The maximum number of indexes that you can create varies by pricing tier. For more information, see Service limits.

URI Parameters

Parameter	Description
service name	Required. Set this to the unique, user-defined name of your search service.
data source name	Required on the URI if using PUT. The name must be lower case, start with a letter or number, have no slashes or dots, and be fewer than 128 characters. The name must start with a letter or number, but the rest of the name can include any letter, number and dashes, as long as the dashes aren't consecutive.
api-version	Required. See API versions for a list of supported versions.

Request Headers

The following table describes the required and optional request headers.

Fields	Description
Content-Type	Required. Set this to `application/json`
api-key	Optional if you're using Azure roles and a bearer token is provided on the request, otherwise a key is required. Create requests must include an `api-key` header set to your admin key (as opposed to a query key). See Connect to Azure AI Search using key authentication for details.

Request Body

The body of the request contains a data source definition, which includes type of the data source, credentials to read the data, as well as an optional data change detection and data deletion detection policies that are used to efficiently identify changed or deleted data in the data source when used with a periodically scheduled indexer.

The following JSON is a high-level representation of the main parts of the definition.

{   
    "name" : (optional on PUT; required on POST) "Name of the data source",  
    "description" : (optional) "Anything you want, or nothing at all",  
    "type" : (required) "Must be a supported data source",
    "credentials" : (required) { "connectionString" : "Connection string for your data source" },
    "container": {
        "name": "Name of the table, view, collection, or blob container you wish to index",
        "query": (optional) 
    },
    "dataChangeDetectionPolicy" : (optional) {See below for details },
    "dataDeletionDetectionPolicy" : (optional) {See below for details },
    "encryptionKey":(optional) { }
}

Request contains the following properties:

Property	Description
name	Required. The name of the data source. A data source name must only contain lowercase letters, digits or dashes, can't start or end with dashes and is limited to 128 characters.
description	An optional description.
type	Required. Must be one of the supported data source types: `azuresql` for Azure SQL Database, Azure SQL Managed Instance or SQL Server instance running on Azure VM `cosmosdb` for the Azure Cosmos DB for NoSQL, Azure Cosmos DB for MongoDB (preview) or Azure Cosmos DB for Apache Gremlin (preview) `azureblob` for Azure Blob Storage `adlsgen2` for Azure Data Lake Storage Gen2 `azuretable` for Azure Table Storage `azurefile` for Azure Files (preview) `mysql` for Azure Database for MySQL (preview) `sharepoint` for SharePoint in Microsoft 365(preview)
credentials	Required. It contains a `connectionString` property that specifies the connection string for the data source. The format of the connection string depends on the data source type: For Azure SQL Database, this is the usual SQL Server connection string. If you're using Azure portal to retrieve the connection string, choose the `ADO.NET connection string` option. For Azure Cosmos DB, the connection string must be in the following format: `"AccountEndpoint=https://[your account name].documents.azure.com;AccountKey=[your account key];Database=[your database id]"`. All of the values are required. You can find them in the Azure portal. For Azure Blob Storage, the connection string formats are defined in the Credentials section of How to configure blob indexing in Azure AI Search. Azure Storage, Azure SQL Database, and Azure Cosmos DB also support a managed identity connection string that doesn't include an account key in the connection string. To use the managed identity connection string format, follow the instructions for Setting up an indexer connection to a data source using a managed identity. If you're updating the data source, the connection string isn't required. The values `<unchanged>` or `<redacted>` can be used in place of the actual connection string.
container	Required. Specifies the data to index using the `name` (required) and `query` (optional) properties: `name`: For Azure SQL, specifies the table or view. You can use schema-qualified names, such as `[dbo].[mytable]`. For Azure Cosmos DB, specifies the SQL API collection. For Azure Blob Storage, specifies the storage container. For Azure Table Storage, specifies the name of the table. `query`: For Azure Cosmos DB, allows you to specify a query that flattens an arbitrary JSON document layout into a flat schema that Azure AI Search can index. For Azure Blob Storage, allows you to specify a virtual folder within the blob container. For example, for blob path `mycontainer/documents/blob.pdf`, `documents` can be used as the virtual folder. For Azure Table Storage, allows you to specify a query that filters the set of rows to be imported. For Azure SQL, query isn't supported. Use views instead.
dataChangeDetectionPolicy	Optional. Used to identify changed data items. Supported policies vary based on the data source type. Valid policies are High Watermark Change Detection Policy and SQL Integrated Change Detection Policy. High Watermark Change Detection Policy depends on an existing column or property that is updated in tandem with other updates (all inserts result in an update to the watermark column), and the change in value is higher. For Cosmos DB data sources, you must use the `_ts` property. For Azure SQL, an indexed `rowversion` column is the ideal candidate for use with the high water mark policy. For Azure Storage, change detection is built-in using lastModified values, eliminating any need to set the dataChangeDetectionPolicy for blob or table storage. SQL Integrated Change Detection Policy is used to reference the native change detection features in SQL Server. This policy can only be used with tables; it can't be used with views. You need to enable change tracking for the table you're using before you can use this policy. See Enable and disable change tracking for instructions. For more information about change detection support in the indexer, see Connect to and index Azure SQL content.
dataDeletionDetectionPolicy	Optional. Used to identify deleted data items. Currently, the only supported policy is the Soft Delete Policy, which identifies deleted items based on the value of a 'soft delete' column or property in the data source. Only columns with string, integer, or boolean values are supported. The value used as `softDeleteMarkerValue` must be a string, even if the corresponding column holds integers or booleans. For example, if the value that appears in your data source is 1, use "1" as the `softDeleteMarkerValue`.
encryptionKey	Optional. Used to encrypt the data source at rest with your own keys, managed in your Azure Key Vault. Available for billable search services created on or after 2019-01-01. An `encryptionKey` section contains a user-defined `keyVaultKeyName` (required), a system-generated `keyVaultKeyVersion` (required), and a `keyVaultUri` providing the key (required, also referred to as DNS name). An example URI might be "https://my-keyvault-name.vault.azure.net". Optionally, you can specify `accessCredentials` if you aren't using a managed system identity. Properties of `accessCredentials` include `applicationId` (Microsoft Entra ID application ID that was granted access permissions to your specified Azure Key Vault), and `applicationSecret` (authentication key of the registered application). An example in the next section illustrates the syntax.

Response

For a successful request: 201 Created.

Examples

Example: Azure SQL with change detection (High Watermark Change Detection Policy)

{   
    "name" : "asqldatasource",  
    "description" : "a description",  
    "type" : "azuresql",  
    "credentials" : { "connectionString" : "Server=tcp:....database.windows.net,1433;Database=...;User ID=...;Password=...;Trusted_Connection=False;Encrypt=True;Connection Timeout=30;" },  
    "container" : { "name" : "sometable" },  
    "dataChangeDetectionPolicy" : { "@odata.type" : "#Microsoft.Azure.Search.HighWaterMarkChangeDetectionPolicy", "highWaterMarkColumnName" : "RowVersion" }
}

Example: Azure SQL with change detection (SQL Integrated Change Tracking Policy)

{   
    "name" : "asqldatasource",  
    "description" : "a description",  
    "type" : "azuresql",  
    "credentials" : { "connectionString" : "Server=tcp:....database.windows.net,1433;Database=...;User ID=...;Password=...;Trusted_Connection=False;Encrypt=True;Connection Timeout=30;" },  
    "container" : { "name" : "sometable" },  
    "dataChangeDetectionPolicy" : { "@odata.type" : "#Microsoft.Azure.Search.SqlIntegratedChangeTrackingPolicy" }
}

Example: Azure SQL with change detection with deletion detection

Recall that the properties for deletion detection are softDeleteColumnName and softDeleteMarkerValue.

{   
    "name" : "asqldatasource",  
    "description" : "a description",  
    "type" : "azuresql",  
    "credentials" : { "connectionString" : "Server=tcp:....database.windows.net,1433;Database=...;User ID=...;Password=...;Trusted_Connection=False;Encrypt=True;Connection Timeout=30;" },  
    "container" : { "name" : "sometable" },   
    "dataDeletionDetectionPolicy" : { "@odata.type" : "#Microsoft.Azure.Search.SoftDeleteColumnDeletionDetectionPolicy", "softDeleteColumnName" : "IsDeleted", "softDeleteMarkerValue" : "true" }  
}

Example: Data source with required properties only

Optional properties related to change and deletion detection can be omitted if you only intend to use the data source for one-time copy of the data:

{   
    "name" : "asqldatasource",  
    "description" : "anything you want, or nothing at all",  
    "type" : "azuresql",  
    "credentials" : { "connectionString" : "Server=tcp:....database.windows.net,1433;Database=...;User ID=...;Password=...;Trusted_Connection=False;Encrypt=True;Connection Timeout=30;" },  
    "container" : { "name" : "sometable" }  
}

Example: Omitting credentials

If you intend to update the data source, the credentials aren't required. The values <unchanged> or <redacted> can be used in place of the connection string.

{
    "name" : "adatasource",
    "description": "a description",
    "type": "azuresql",
    "credentials": { "connectionString": "<unchanged>" },
    "container" : { "name": "sometable" }
}

Example: Encryption keys

Encryption keys are customer-managed keys used for extra encryption. For more information, see Encryption using customer-managed keys in Azure Key Vault.

{
    "name" : "adatasource",
    "description": "a description",
    "type": "azuresql",
    "credentials": { "connectionString": "<unchanged>" },
    "container" : { "name": "sometable" }
    "encryptionKey": (optional) { 
      "keyVaultKeyName": "Name of the Azure Key Vault key used for encryption",
      "keyVaultKeyVersion": "Version of the Azure Key Vault key",
      "keyVaultUri": "URI of Azure Key Vault, also referred to as DNS name, that provides the key. An example URI might be https://my-keyvault-name.vault.azure.net",
      "accessCredentials": (optional, only if not using managed system identity) {
        "applicationId": "Microsoft Entra ID application ID that was granted access permissions to your specified Azure Key Vault",
        "applicationSecret": "Authentication key of the registered application)"}
      }
}

Share via