How to write a string from a data factory pipeline to an azure table storage

Question

Hello,

I have a pipeline in Azure Data Factory. This pipeline gets a string as an input value. I now want to write this string to an Azure Table Storage.

I am converting the string to the following JSON-schema:

{

        "PartitionKey": <>,

        "RowKey": <>,

        'state': <>,

        'body': <>

    }

I have also created a system-assignes managed identity to the Data Factory and given it contributor rights to the storage account.

I have tried using a web-activity to send a REST-API to the table storage but there I am still getting authorization issues even though the data factory has sufficient rights.

I have also tried a copy activity but that doesn't allow single values or JSON-objects as a source.

How would I best go about solving this?

And please don't just answer with paragraphs copied straight from Chat GPT. I have tried finding a solution with that for hours now, so it will not give any new input.

Answer 1

Hello Benedikt !

Thank you for posting on Microsoft Learn.

ADF doesn't natively support writing JSON to Table Storage but if you are comfortable with Azure functions you can create a HTTP trigger which supports upsert, and accepts a JSON POST body.

import logging
import os
from azure.data.tables import TableServiceClient
import azure.functions as func
import json
connection_string = os.environ["AzureWebJobsStorage"]
table_name = os.environ["TableName"]
def main(req: func.HttpRequest) -> func.HttpResponse:
    try:
        data = req.get_json()
        table_client = TableServiceClient.from_connection_string(conn_str=connection_string)
        table = table_client.get_table_client(table_name=table_name)
        entity = {
            "PartitionKey": data["PartitionKey"],
            "RowKey": data["RowKey"],
            "state": data["state"],
            "body": data["body"]
        }
        table.upsert_entity(entity)
        return func.HttpResponse("Success", status_code=200)
    except Exception as e:
        logging.error(str(e))
        return func.HttpResponse("Error: " + str(e), status_code=500)

On the Azure Function, you need to enable Azure AD authentication and set it to accept only requests from authenticated users (disable anonymous).

Then assign System-Assigned Managed Identity a role in your Function App (for example Function App Contributor or custom RBAC that allows Microsoft.Web/sites/functions/invoke/action).

Then last step in the ADF Pipeline, use a Web Activity with the following configuration :

• URL: https://<functionapp>.azurewebsites.net/api/<function_name>
• Method: POST
• Authentication: MSI
• Resource: https://<functionapp>.azurewebsites.net
• Body: pass your JSON string (with PartitionKey, RowKey...)

{
  "PartitionKey": "Logs",
  "RowKey": "20250607-123",
  "state": "Success",
  "body": "Pipeline completed successfully"
}