How to update data into Azure storage table from datafactory?

THIMMAIAH GARI,PRASHANTH,, 201 Reputation points
2020-10-13T12:07:05.143+00:00

1) Can i use storage table for capture log details of pipeline via datafactory. will it allow insert new values in
table?

2) I want to update specific column in storage table. how do i update the column which activity I should
use?

like I want to create table "control_table" with columns having "table_name" and "last_update" columns. every time I load data , i want to update the "last update column".

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,273 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,100 questions
0 comments No comments
{count} votes

Accepted answer
  1. HarithaMaddi-MSFT 10,126 Reputation points
    2020-10-14T15:33:41.837+00:00

    Hi @THIMMAIAH GARI,PRASHANTH,, ,

    Welcome to Microsoft Q&A Platform. Thanks for posting the query.

    Yes, Azure table storage can be used to log the details from Azure data factory. Both insert and update are possible.

    • I have used copy activity to implement the above requirement with dummy source file whereas additional columns will be used for populating table name and date that can come from pipeline variables. This activity can be used after table is loaded in ADF pipeline to record the load into the Azure table.
    • Replace will replace entire property of previous entity with new entity if the entity exists. If the new entity doesn't define property or having property value null then that property will be removed on updating whereas in merge, old properties will be retained even if the new entity didn't define new properties in new entity and it updates if new values are defined for old properties if entity exists. Hence, I have used merge in this scenario.
    • Partition key + Row key combination has to be unique in a table and hence, partition key can be used appropriately from your scenario. I have used static value of 1 for partition key and used table name as row key in this scenario. So that whenever date property changes for same table name under 1 partition id.

    Please find below GIF and JSON attached of the pipeline.

    32308-tablestorageadfupsert.gif
    32392-tablestorageupsertjson.txt

    Ref: difference-between-insert-or-merge-entity-and-insert-or-replace-entity

    Hope this helps! Please let us know if it does not align with requirement or for further queries and we will be glad to assist.

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful