How do I check if azure table storage exist in data factory

Question

Hi, I have pipeline that copy azure table storage, I inject parameters to this pipeline (for the linked service and dataset). One of the parameters could be source table name that doesn't exist, I tried to use this

"azureTableSourceIgnoreTableNotFound": true

but it doesn't work and I still get an exception when the table is not exist. So, I want to check if the table is exist before the copy activity because I don't want my pipeline to fall if the table is not exist. I didn't find a way to do it. Please assist

Answer 1

Hi again start by Creating an Azure Function (e.g., in Python or C#) that checks whether the specified table exists in Azure Table Storage. The function can use the Azure Storage SDK to interact with Table Storage. fine below example of using Azure SDK for Python

from azure.cosmosdb.table.tableservice import TableService  def check_table_existence(account_name, account_key, table_name):     table_service = TableService(account_name=account_name, account_key=account_key)      try:         table_service.get_table_service_properties(table_name)         return True     except Exception as e:         if 'TableNotFound' in str(e):             return False         else:             raise  # Use the check_table_existence function in your Azure Function 

and within your ADF pipeline, use an Azure Function activity to invoke the Azure Function createdearlier and Pass the required parameters like (account name, account key, table name) to the function. finally capture the output of the Azure Function activity in a variable or use it in a subsequent activity (e.g., a conditional activity) to decide whether to proceed with the copy activity.

eg snippet to illustrate the structure

{     "name": "CheckTableExistence",     "type": "AzureFunction",     "linkedServiceName": {         "referenceName": "AzureFunctionLinkedService",         "type": "LinkedServiceReference"     },     "typeProperties": {         "functionName": "CheckTableExistence",         "method": "GET",         "body": {             "accountName": "<StorageAccountName>",             "accountKey": "<StorageAccountKey>",             "tableName": "@pipeline().parameters.azureTableSourceName"         },         "authKey": {             "name": "AzureWebJobsStorage",             "type": "LinkedServiceReference"         }     },     "outputs": [         {             "name": "output",             "linkedServiceName": {                 "referenceName": "AzureBlobStorageLinkedService",                 "type": "LinkedServiceReference"             },             "parameters": {                 "statusCode": "@activity('CheckTableExistence').output.statusCode",                 "body": "@activity('CheckTableExistence').output.body"             }         }     ] } 

For more details I suggest you to run through the documentation let me give you the link below https://learn.microsoft.com/en-us/search/?scope=Intune&terms=Azure%20Function If this helps kindly accept the answer thanks much

Answer 2

Hi OGINTZ Marina, Thank you for posting query in Microsoft Q&A Platform.

You can consider using Lookup activity. If Lookup activity fails, that means table does not exist and on failure of it connect a wait activity. This avoids pipeline failure.On success of lookup activity, that means when table exists, we can run copy activity.

Hope this helps. Please let me know if any further queries.

---Please consider hitting Accept Answer button. Accepted answers help community as well. Thank you.