Data connector reference for the Codeless Connector Platform

Article
12/19/2023

To create a data connector with the Codeless Connector Platform (CCP), use this document as a supplement to the Microsoft Sentinel REST API for Data Connectors reference docs. Specifically this reference document expands on the following details:

The data connector kind, RestApiPoller, which is used for the CCP.
Authorization configuration
Data source request and response configuration options
Data stream paging options
Data collection rule map

Each dataConnector represents a specific connection of a Microsoft Sentinel data connector. One data connector might have multiple connections, which fetch data from different endpoints. The JSON configuration built using this reference document is used to complete the deployment template for the CCP data connector.

For more information, see Create a codeless connector for Microsoft Sentinel.

Data Connectors - Create or update

Reference the Create or Update operation in the REST API docs to find the latest stable or preview API version. The difference between the create and the update operation is the update requires the etag value.

PUT method

https://management.azure.com/subscriptions/{{subscriptionId}}/resourceGroups/{{resourceGroupName}}/providers/Microsoft.OperationalInsights/workspaces/{{workspaceName}}/providers/Microsoft.SecurityInsights/dataConnectors/{{dataConnectorId}}?api-version={{apiVersion}}

URI parameters

For more information about the latest API version, see Data Connectors - Create or Update URI Parameters.

Name	Description
dataConnectorId	The data connector ID must be a unique name and is the same as the `name` parameter in the request body.
resourceGroupName	The name of the resource group, not case sensitive.
subscriptionId	The ID of the target subscription.
workspaceName	The name of the workspace, not the ID. Regex pattern: `^[A-Za-z0-9][A-Za-z0-9-]+[A-Za-z0-9]$`
api-version	The API version to use for this operation.

Request body

The request body for the CCP data connector has the following structure:

{
   "name": "{{dataConnectorId}}",
   "kind": "RestApiPoller",
   "etag": "",
   "properties": {
        "connectorDefinitionName": "",
        "auth": {},
        "request": {},
        "response": {},
        "paging": "",
        "dcrConfig": ""
   }
}

RestApiPoller represents the codeless API Poller connector.

Name	Required	Type	Description
name	True	string	The unique name of the connection matching the URI parameter
kind	True	string	Must be `RestApiPoller`
etag		GUID	Leave empty for creation of new connectors. For update operations, the etag must match the existing connector's etag (GUID).
properties.connectorDefinitionName		string	The name of the DataConnectorDefinition resource that defines the UI configuration of the data connector. For more information, see Data Connector Definition.
properties.auth	True	Nested JSON	Describes the authentication properties for polling the data. For more information, see authentication configuration.
properties.request	True	Nested JSON	Describes the request payload for polling the data, such as the API endpoint. For more information, see request configuration.
properties.response	True	Nested JSON	Describes the response object and nested message returned from the API when polling the data. For more information, see response configuration.
properties.paging		Nested JSON	Describes the pagination payload when polling the data. For more information, see paging configuration.
properties.dcrConfig		Nested JSON	Required parameters when the data is sent to a Data Collection Rule (DCR). For more information, see DCR configuration.

Authentication configuration

The CCP supports the following authentication types:

Basic
APIKey
OAuth2

Note

CCP OAuth2 implementation does not support certificate credentials.

As a best practice, use parameters in the auth section instead of hard-coding credentials. For more information, see Secure confidential input.

In order to create the deployment template which also uses parameters, you need to escape the parameters in this section with an extra starting [. This allows the parameters to assign a value based on the user interaction with the connector. For more information, see Template expressions escape characters.

To enable the credentials to be entered from the UI, the connectorUIConfig section requires instructions with the desired parameters. For more information, see Data connector definitions reference for the Codeless Connector Platform.

Basic auth

Field	Required	Type
UserName	True	string
Password	True	string

Example Basic auth using parameters defined in connectorUIconfig:

"auth": {
    "type": "Basic",
    "UserName": "[[parameters('username')]",
    "Password": "[[parameters('password')]"
}

APIKey

Field	Required	Type	Description	Default value
ApiKey	True	string	user secret key
ApiKeyName		string	name of the Uri header containing the ApiKey value	`Authorization`
ApiKeyIdentifier		string	string value to prepend the token	`token`
IsApiKeyInPostPayload		boolean	send secret in POST body instead of header	`false`

APIKey auth examples:

"auth": {
    "type": "APIKey",
    "ApiKey": "[[parameters('apikey')]",
    "ApiKeyName": "X-MyApp-Auth-Header",
    "ApiKeyIdentifier": "Bearer"
}

This example results in the secret defined from user input sent in the following header: X-MyApp-Auth-Header: Bearer apikey

"auth": { 
    "type": "APIKey",
    "ApiKey": "123123123",
}

This example uses the default values and results in the following header: Authorization: token 123123123

"auth": { 
    "type": "APIKey",
    "ApiKey": "123123123",
    "ApiKeyName": ""
}

Since the ApiKeyName is explicitly set to "", the result is the following header: Authorization: 123123123

OAuth2

The Codeless Connector Platform supports OAuth 2.0 authorization code grant and client credentials. The Authorization Code grant type is used by confidential and public clients to exchange an authorization code for an access token. After the user returns to the client via the redirect URL, the application will get the authorization code from the URL and use it to request an access token.

Field	Required	Type	Description
ClientId	True	String	The client id
ClientSecret	True	String	The client secret
AuthorizationCode	True when grantType = `authorization_code`	String	If grant type is `authorization_code` this field value will be the authorization code returned from the auth serve.
Scope	True for `authorization_code` grant type optional for `client_credentials` grant type	String	A space-separated list of scopes for user consent. For more information, see OAuth2 scopes and permissions.
RedirectUri	True when grantType = `authorization_code`	String	URL for redirect, must be `https://portal.azure.com/TokenAuthorize/ExtensionName/Microsoft_Azure_Security_Insights`
GrantType	True	String	`authorization_code` or `client_credentials`
TokenEndpoint	True	String	URL to exchange code with valid token in `authorization_code` grant or client id and secret with valid token in `client_credentials` grant.
TokenEndpointHeaders		Object	An optional key value object to send custom headers to token server
TokenEndpointQueryParameters		Object	An optional key value object to send custom query params to token server
AuthorizationEndpoint	True	String	URL for user consent for `authorization_code` flow
AuthorizationEndpointHeaders		Object	An optional key value object to send custom headers to auth server
AuthorizationEndpointQueryParameters		Object	An optional key value pair used in OAuth2 authorization code flow request

Auth code flow is for fetching data on behalf of a user's permissions and client credentials is for fetching data with application permissions. The data server grants access to the application. Since there is no user in client credentials flow, no authorization endpoint is needed, only a token endpoint.

Example: OAuth2 auth code grant

"auth": {
    "type": "OAuth2",
    "ClientId": "[parameters('appId')]",
    "ClientSecret": "[parameters('appSecret')]",
    "tokenEndpoint": "https://login.microsoftonline.com/{{tenantId}}/oauth2/v2.0/token",
    "authorizationEndpoint": "https://login.microsoftonline.com/{{tenantId}}/oauth2/v2.0/authorize",
    "authorizationEndpointHeaders": {},
    "authorizationEndpointQueryParameters": {
        "prompt": "consent"
    },
    "redirectUri": "https://portal.azure.com/TokenAuthorize/ExtensionName/Microsoft_Azure_Security_Insights",
    "tokenEndpointHeaders": {
        "Accept": "application/json",
        "Content-Type": "application/x-www-form-urlencoded"
    },
    "TokenEndpointQueryParameters": {},
    "scope": "openid offline_access some_scope",
    "grantType": "authorization_code"
}

Example:

"auth": {
    "type": "OAuth2",
    "ClientId": "[parameters('appId')]",
    "ClientSecret": "[parameters('appSecret')]",
    "tokenEndpoint": "https://login.microsoftonline.com/{{tenantId}}/oauth2/v2.0/token",
    "tokenEndpointHeaders": {
        "Accept": "application/json",
        "Content-Type": "application/x-www-form-urlencoded"
    },
    "TokenEndpointQueryParameters": {},
    "scope": "openid offline_access some_scope",
    "grantType": "client_credentials"
}

Request configuration

The request section defines how the CCP data connector sends requests to your data source, like the API endpoint and how often to poll that endpoint.

Field	Required	Type	Description
ApiEndpoint	True	String	URL for remote server. Defines the endpoint to pull data from.
RateLimitQPS		Integer	Defines the number of calls or queries allowed in a second.
QueryWindowInMin		Integer	Defines the available query window in minutes. Minimum is 1 minute. Default is 5 minutes.
HttpMethod		String	Defines the API method: `GET`(default) or `POST`
QueryTimeFormat		String	Defines the date and time format the endpoint (remote server) expects. The CCP uses the current date and time wherever this variable is used. Possible values are the constants: `UnixTimestamp`, `UnixTimestampInMills` or any other valid representation of date time, for example: `yyyy-MM-dd`, `MM/dd/yyyy HH:mm:ss` default is ISO 8601 UTC
RetryCount		Integer (1...6)	Defines `1` to `6` retries allowed to recover from a failure. Default is `3`.
TimeoutInSeconds		Integer (1...180)	Defines the request timeout, in seconds. Default is `20`
IsPostPayloadJson		Boolean	Determines whether the POST payload is in JSON format. Default is `false`
Headers		Object	Key value pairs that define the request headers.
QueryParameters		Object	Key value pairs that define the request query parameters.
StartTimeAttributeName	True when `EndTimeAttributeName` is set	String	Defines the query parameter name for query start time. See example.
EndTimeAttributeName	True when `StartTimeAttributeName` is set	String	Defines the query parameter name for query end time.
QueryTimeIntervalAttributeName		String	If the endpoint requires a specialized format for querying the data on a time frame, then use this property with the `QueryTimeIntervalPrepend` and the `QueryTimeIntervalDelimiter` parameters. See example.
QueryTimeIntervalPrepend	True when `QueryTimeIntervalAttributeName` is set	String	See `QueryTimeIntervalAttributeName`
QueryTimeIntervalDelimiter	True when `QueryTimeIntervalAttributeName` is set	String	See `QueryTimeIntervalAttributeName`
QueryParametersTemplate		String	Query template to use when passing parameters in advanced scenarios. br>For example: `"queryParametersTemplate": "{'cid': 1234567, 'cmd': 'reporting', 'format': 'siem', 'data': { 'from': '{_QueryWindowStartTime}', 'to': '{_QueryWindowEndTime}'}, '{_APIKeyName}': '{_APIKey}'}"`

When the API requires complex parameters, use the queryParameters or queryParametersTemplate which include some built-in variables.

built-in variable	for use in `queryParameters`	for use in `queryParametersTemplate`
`_QueryWindowStartTime`	yes	yes
`_QueryWindowEndTime`	yes	yes
`_APIKeyName`	no	yes
`_APIKey`	no	yes

StartTimeAttributeName example

Consider this example:

StartTimeAttributeName = from
EndTimeAttributeName = until
ApiEndpoint = https://www.example.com

The query sent to the remote server is: https://www.example.com?from={QueryTimeFormat}&until={QueryTimeFormat + QueryWindowInMin}

QueryTimeIntervalAttributeName example

Consider this example:

QueryTimeIntervalAttributeName = interval
QueryTimeIntervalPrepend = time:
QueryTimeIntervalDelimiter = ..
ApiEndpoint = https://www.example.com

The query sent to the remote server is: https://www.example.com?interval=time:{QueryTimeFormat}..{QueryTimeFormat + QueryWindowInMin}

Request examples using Microsoft Graph as data source API

This example queries messages with a filter query parameter. For more information, see Microsoft Graph API query parameters.

"request": {
  "apiEndpoint": "https://graph.microsoft.com/v1.0/me/messages",
  "httpMethod": "Get",
  "queryTimeFormat": "yyyy-MM-ddTHH:mm:ssZ",
  "queryWindowInMin": 10,
  "retryCount": 3,
  "rateLimitQPS": 20,
  "headers": {
    "Accept": "application/json",
    "User-Agent": "Example-app-agent"
  },
  "QueryTimeIntervalAttributeName": "filter",
  "QueryTimeIntervalPrepend": "receivedDateTime gt ",
  "QueryTimeIntervalDelimiter": " and receivedDateTime lt "
}

The previous example sends a GET request to https://graph.microsoft.com/v1.0/me/messages?filter=receivedDateTime gt {time of request} and receivedDateTime lt 2019-09-01T17:00:00.0000000. The timestamp updates for each queryWindowInMin time.

The same results are achieved with this example:

"request": {
  "apiEndpoint": "https://graph.microsoft.com/v1.0/me/messages",
  "httpMethod": "Get",
  "queryTimeFormat": "yyyy-MM-ddTHH:mm:ssZ",
  "queryWindowInMin": 10,
  "retryCount": 3,
  "rateLimitQPS": 20,
  "headers": {
    "Accept": "application/json",
  },
  "queryParameters": {
    "filter": "receivedDateTime gt {_QueryWindowStartTime} and receivedDateTime lt {_QueryWindowEndTime}"
  }
}

Another option is when the data source expects 2 query parameters, one for start time and one for end time.

Example:

"request": {
  "apiEndpoint": "https://graph.microsoft.com/v1.0/me/calendarView",
  "httpMethod": "Get",
  "queryTimeFormat": "yyyy-MM-ddTHH:mm:ssZ",
  "queryWindowInMin": 10,
  "retryCount": 3,
  "rateLimitQPS": 20,
  "headers": {
    "Accept": "application/json",
  },
  "StartTimeAttributeName": "startDateTime",
  "EndTimeAttributeName": "endDateTime",
}

This sends a GET request to https://graph.microsoft.com/me/calendarView?startDateTime=2019-09-01T09:00:00.0000000&endDateTime=2019-09-01T17:00:00.0000000

For complex queries, use QueryParametersTemplate. This next example sends a POST request with parameters in the body.

Example:

request: {
  "apiEndpoint": "https://graph.microsoft.com/v1.0/me/messages",
  "httpMethod": "POST",
  "queryTimeFormat": "yyyy-MM-ddTHH:mm:ssZ",
  "queryWindowInMin": 10,
  "retryCount": 3,
  "rateLimitQPS": 20,
  "headers": {
    "Accept": "application/json",
  },
  "isPostPayloadJson": true,
  "queryParametersTemplate": "{\"query":"TableName | where createdTimestamp between (datetime({_QueryWindowStartTime}) .. datetime({_QueryWindowEndTime}))\"}"
}

Response configuration

Define the response handling of your data connector with the following parameters:

Field	Required	Type	Description
EventsJsonPaths	True	List of Strings	Defines the path to the message in the response JSON. A JSON path expression specifies a path to an element, or a set of elements, in a JSON structure
SuccessStatusJsonPath		String	Defines the path to the success message in the response JSON. When this parameter is defined, the `SuccessStatusValue` parameter should also be defined.
SuccessStatusValue		String	Defines the path to the success message value in the response JSON
IsGzipCompressed		Boolean	Determines whether the response is compressed in a gzip file
format	True	String	`json` or `csv` or `xml`
CompressionAlgo		String	The compressions algorithm, either `multi-gzip` or `deflate`. For gzip compression algorithm, just configure `IsGzipCompressed` to `True` instead of setting a value for this parameter.
CsvDelimiter		String	If response format is CSV and you want to change the default CSV delimiter of `","`
HasCsvBoundary		Boolean	Indicate if CSV data has a boundary
HasCsvHeader		Boolean	Indicate if CSV data has a header, default is `True`
CsvEscape		String	Escape character for a field boundary, default is `"` For example, a CSV with headers `id,name,avg` and a row of data containing spaces like `1,"my name",5.5` requires the `"` field boundary.
ConvertChildPropertiesToArray		Boolean	Special case in which the remote server returns an object instead of a list of events where each property has data in it.

Note

CSV format type is parsed by the RFC4180 specification.

Response configuration examples

A server response with JSON format is expected, with the requested data in the property value. The response property status indicates to ingest the data only if the value is success.

"response": {
  "EventsJsonPaths ": ["$.value"],
  "format": "json",
  "SuccessStatusJsonPath": "$.status",
  "SuccessStatusValue": "success",
  "IsGzipCompressed: true
 }

The expected response in this example prepares for a CSV with no header.

"response": {
  "EventsJsonPaths ": ["$"],
  "format": "csv",
  "HasCsvHeader": false
 }

Paging configuration

When the data source can't send the entire response payload all at once, the CCP data connector needs to know how to receive portions of the data in response pages. The paging types to choose from are:

Paging type	decision factor
LinkHeader PersistentLinkHeader NextPageUrl	Does the API response have links to next and previous pages?
NextPageToken PersistentToken	Does the API response have a token or cursor for the next and previous pages?
Offset	Does the API response support a parameter for the number of objects to skip when paging?

Configure LinkHeader or PersistentLinkHeader

The most common paging type is when a server data source API provides URLs to the next and previous pages of data. For more information on the Link Header specification, see RFC 5988.

LinkHeader paging means the API response includes either:

the Link HTTP response header
or a JSON path to retrieve the link from the response body.

PersistentLinkHeader paging has the same properties as LinkHeader, except the link header persists in backend storage. This option enables paging links across query windows. For example, some APIs don't support query start times or end times. Instead, they support a server side cursor. Persistent page types can be used to remember the server side cursor. For more information, see What is a cursor?.

Note

There can be only one query running for the connector with PersistentLinkHeader to avoid race conditions on the server side cursor. This may affect latency.

Field	Required	Type	Description
LinkHeaderTokenJsonPath	False	String	Use this property to indicate where to get the value in the response body. For example, if the data source returns the following JSON: `{ nextPage: "foo", value: [{data}]}` then `LinkHeaderTokenJsonPath` will be `$.nextPage`
PageSize	False	Integer	How many events per page
PageSizeParameterName	False	String	Query parameter name for the page size

Here are some examples:

Paging: {
  "pagingType": "LinkHeader",
  "linkHeaderTokenJsonPath" : "$.metadata.links.next"
}

Paging: {
 "pagingType" : "PersistentLinkHeader", 
 "pageSizeParameterName" : "limit", 
 "pageSize" : 500 
}

Configure NextPageUrl

NextPageUrl paging means the API response includes a complex link in the response body similar to LinkHeader, but the URL is included in the response body instead of the header.

Field	Required	Type	Description
PageSize	False	Integer	How many events per page
PageSizeParameterName	False	String	Query parameter name for the page size
NextPageUrl	False	String	Only if the connector is for Coralogix API
NextPageUrlQueryParameters	False	Object Key value pairs – adding custom query parameter to each request for the next page
NextPageParaName	False	String	Determines the next page name in the request.
HasNextFlagJsonPath	False	String	Defines the path to the HasNextPage flag attribute
NextPageRequestHeader	False	String	Determines the next page header name in the request.
NextPageUrlQueryParametersTemplate	False	String	Only if the connector is for Coralogix API

Example:

Paging: {
 "pagingType" : "NextPageUrl", 
  "nextPageTokenJsonPath" : "$.data.repository.pageInfo.endCursor", 
  "hasNextFlagJsonPath" : "$.data.repository.pageInfo.hasNextPage", 
  "nextPageUrl" : "https://api.github.com/graphql", 
  "nextPageUrlQueryParametersTemplate" : "{'query':'query{repository(owner:\"xyz\")}" 
}

Configure NextPageToken or PersistentToken

NextPageToken pagination uses a token (a hash or a cursor) that represents the state of the current page. The token is included in the API response, and the client appends it to the next request to fetch the next page. This method is often used when the server needs to maintain the exact state between requests.

PersistentToken pagination uses a token that persists server side. The server remembers the last token retrieved by the client and provides the next token in subsequent requests. The client continues where it left off even if it makes new requests later.

Field	Required	Type	Description
PageSize	False	Integer	How many events per page
PageSizeParameterName	False	string	Query parameter name for the page size
NextPageTokenJsonPath	False	string	JSON path for next page token in the response body.
NextPageTokenResponseHeader	False	string	If `NextPageTokenJsonPath` is empty, use the token is in this header name for the next page.
NextPageParaName	False	string	Determines the next page name in the request.
HasNextFlagJsonPath	False	string	Defines the path to a HasNextPage flag attribute when determining if more pages are left in the response.
NextPageRequestHeader	False	string	Determines the next page header name in the request.

Examples:

Paging: {
 "pagingType" : "NextPageToken", 
 "nextPageRequestHeader" : "ETag", 
 "nextPageTokenResponseHeader" : "ETag" 
}

Paging: {
 "pagingType" : "PersistentToken", 
    "nextPageParaName" : "gta", 
    "nextPageTokenJsonPath" : "$.alerts[-1:]._id" 
}

Configure Offset

Offset pagination specifies the number of pages to skip and a limit on the number of events to retrieve per page in the request. Clients fetch a specific range of items from the data set.

Field	Required	Type	Description
PageSize	False	Integer	How many events per page
PageSizeParameterName	False	String	Query parameter name for the page size
OffsetParaName	False	String	The next request query parameter name. The CCP calculates the offset value for each request, (all events ingested + 1)

Example:

Paging: {
   
       "pagingType": "Offset", 
        "offsetParaName": "offset" 
 }

DCR configuration

Field	Required	Type	Description
DataCollectionEndpoint	True	String	DCE (Data Collection Endpoint) for example: `https://example.ingest.monitor.azure.com`.
DataCollectionRuleImmutableId	True	String	The DCR immutable ID. Find it by viewing the DCR creation response or using the DCR API
StreamName	True	string	This value is the `streamDeclaration` defined in the DCR (prefix must begin with Custom-)

Example CCP data connector

Here's an example of all the components of the CCP data connector JSON together.

{
   "kind": "RestApiPoller",
   "properties": {
      "connectorDefinitionName": "ConnectorDefinitionExample",
      "dcrConfig": {
           "streamName": "Custom-ExampleConnectorInput",
           "dataCollectionEndpoint": "https://example-dce-sbsr.location.ingest.monitor.azure.com",
           "dataCollectionRuleImmutableId": "dcr-32_character_hexadecimal_id"
            },
      "dataType": "ExampleLogs",
      "auth": {
         "type": "Basic",
         "password": "[parameters('username')]",
         "userName": "[parameters('password')]"
      },
      "request": {
         "apiEndpoint": "https://rest.contoso.com/example",
         "rateLimitQPS": 10,
         "queryWindowInMin": 5,
         "httpMethod": "GET",
         "queryTimeFormat": "UnixTimestamp",
         "startTimeAttributeName": "t0",
         "endTimeAttributeName": "t1",
         "retryCount": 3,
         "timeoutInSeconds": 60,
         "headers": {
            "Accept": "application/json",
            "User-Agent": "Example-app-agent"
         } 
      },
      "paging": {
         "pagingType": "LinkHeader"
      },
      "response": {
         "eventsJsonPaths": ["$"]
      }
   }
}

Data connector reference for the Codeless Connector Platform

Data Connectors - Create or update

URI parameters

Request body

Authentication configuration

Basic auth

APIKey

OAuth2

Request configuration

StartTimeAttributeName example

QueryTimeIntervalAttributeName example

Request examples using Microsoft Graph as data source API

Response configuration

Response configuration examples

Paging configuration

Configure LinkHeader or PersistentLinkHeader

Configure NextPageUrl

Configure NextPageToken or PersistentToken

Configure Offset

DCR configuration

Example CCP data connector

Feedback

Additional resources