Configure Azure Table Storage in a copy activity

This article outlines how to use the copy activity in data pipeline to copy data from and to Azure Table Storage.

Supported configuration

For the configuration of each tab under copy activity, go to the following sections respectively.

• General
• Source
• Destination
• Mapping
• Settings

General

Refer to the General settings guidance to configure the General settings tab.

Source

Go to Source tab to configure your copy activity source. See the following content for the detailed configuration.

The following three properties are required:

• Data store type: Select External.

• Connection: Select an Azure Table Storage connection from the connection list. If no connection exists, then create a new Azure Table Storage connection by selecting New.

• Use query: Specify way to read data. Select Table to read data from the specified table or select Query to read data using queries.

  If you select Table:

  

  • Table: Specify the name of the table in the Azure Table Storage database instance. Select the table from the drop-down list or enter the name manually by selecting Edit.

  If you select Query:

  

  • Table: Specify the name of the table in the Azure Table Storage database instance. Select the table from the drop-down list or enter the name manually by selecting Edit.

  • Query: Specify the custom Table storage query to read data. The source query is a direct map from the $filter query option supported by Azure Table Storage, learn more about the syntax from this article.

    Note

    Azure Table query operation times out in 30 seconds as enforced by Azure Table service. Learn how to optimize the query from Design for querying article.

Under Advanced, you can specify the following fields:

• Ignore table not found: Specifies whether to allow the exception of the table to not exist. It is unselected by default.

• Additional columns: Add additional data columns to store source files' relative path or static value. Expression is supported for the latter.

Destination

Go to Destination tab to configure your copy activity destination. See the following content for the detailed configuration.

The following three properties are required:

• Data store type: Select External.
• Connection: Select an Azure Table Storage connection from the connection list. If no connection exists, then create a new Azure Table Storage connection by selecting New.
• Table: Specify the name of the table in the Azure Table Storage database instance. Select the table from the drop-down list or enter the name manually by selecting Edit.

Under Advanced, you can specify the following fields:

• Insert type: Select the mode to insert data into Azure Table. The modes are Merge and Replace. This property controls whether existing rows in the output table with matching partition and row keys have their values replaced or merged. This setting applies at the row level not the table level. Neither option deletes rows in the output table that don't exist in the input. To learn about how the merge and replace settings work, see Insert or merge entity and Insert or replace entity.

• Partition key value selection: Select from Specify partition value or Use destination column. Partition key value can be a fixed value or it can take value from a destination column.

  If you select Specify partition value:

  • Default partition value: Specify the default partition key value that can be used by the destination.

  If you select Use destination column:

  • Partition key column: Select the name of the column whose column values are used as the partition key. If not specified, "AzureTableDefaultPartitionKeyValue" is used as the partition key.

• Row key value selection: Select from Unique identifier or Use destination column. Row key value can be an auto generated unique identifier or it can take value from a destination column.

  If you select Use destination column:

  • Row key column: Select the name of the column whose column values are used as the row key. If not specified, use a GUID for each row.

• Write batch size: Inserts data into Azure Table when the specified write batch size is hit. Allowed values are integer (number of rows). The default value is 10,000.

• Write batch timeout: Inserts data into Azure Table when the specified write batch timeout is hit. The allowed value is timespan.

• Max concurrent connections: The upper limit of concurrent connections established to the data store during the activity run. Specify a value only when you want to limit concurrent connections

Mapping

For Mapping tab configuration, see Configure your mappings under mapping tab.

Settings

For Settings tab configuration, go to Configure your other settings under settings tab.

Table summary

The following table contains more information about the copy activity in Azure Table Storage.

Source information

Name	Description	Value	Required	JSON script property
Data store type	Your data store type.	External	Yes	/
Connection	Your connection to the source data store.	< your Azure Table Storage connection >	Yes	connection
Use query	The way to read data. Apply Table to read data from the specified table or apply Query to read data using queries.	• Table • Query	Yes	/
Table	The name of the table in the Azure Table Storage database instance.	< your table name >	Yes	tableName
Query	Specify the custom Table storage query to read data. The source query is a direct map from the $filter query option supported by Azure Table Storage, learn more about the syntax from this article.	< your query >	No	azureTableSourceQuery
Ignore table not found	Indicates whether to allow the exception of the table to not exist.	selected or unselected (default)	No	azureTableSourceIgnoreTableNotFound: true or false (default)
Additional columns	Add additional data columns to store source files' relative path or static value. Expression is supported for the latter.	• Name • Value	No	additionalColumns: • name • value

Destination information

Name	Description	Value	Required	JSON script property
Data store type	Your data store type.	External	Yes	/
Connection	Your connection to the destination data store.	< your Azure Table Storage connection >	Yes	connection
Table	The name of the table in the Azure Table Storage database instance.	< your table name >	Yes	tableName
Insert type	The mode to insert data into Azure Table. This property controls whether existing rows in the output table with matching partition and row keys have their values replaced or merged.	• Merge • Replace	No	azureTableInsertType: • merge • replace
Partition key value selection	Partition key value can be a fixed value or it can take value from a destination column.	• Specify partition value • Use destination column	No	/
Default partition value	The default partition key value that can be used by the destination	< your default partition value >	No	azureTableDefaultPartitionKeyValue
Partition key column	The name of the column whose values are used as partition keys. If not specified, "AzureTableDefaultPartitionKeyValue" is used as the partition key.	< your partition key column >	No	azureTablePartitionKeyName
Row key value selection	Row key value can be an auto generated unique identifier or it can take value from a destination column.	• Unique identifier • Use destination column	No	/
Row key column	The name of the column whose column values are used as the row key. If not specified, use a GUID for each row.	< your row key column >	No	azureTableRowKeyName
Write batch size	Inserts data into Azure Table when the write batch size is hit.	integer (default is 10,000)	No	writeBatchSize
Write batch timeout	Inserts data into Azure Table when the write batch timeout is hit	timespan	No	writeBatchTimeout
Max concurrent connections	The upper limit of concurrent connections established to the data store during the activity run. Specify a value only when you want to limit concurrent connections.	< your max concurrent connections >	No	maxConcurrentConnections

Related content

• Azure Table Storage connector overview