Azure Table Writes and Updates

Question

Azure Table Writes and Updates

Sri Madhu 0

Hi Team , I have few columns along with two audit columns in azure table like refresh type and last modified , once the full load completes in adf , I need to update the refresh type to Incremental and update the last modified , can someone help if this is possible.

@KranthiPakala-MSFT has been answered in one question but I was facing issue like duplicate columns were present even there is single column , can you please help me with the same.

Thanks in advance

Anonymous

2025-06-04T11:07:33.7933333+00:00
@Sri Madhu
If you're using ADF for the full load, add a Web Activity at the end of your pipeline that:

Calls an Azure Function or REST API endpoint.

Updates the RefreshType to "Incremental" and updates the LastModified timestamp.

To assist more on this issue please help me with below questions

How are you currently retrieving and updating entities in Azure Table Storage? (e.g., Azure Data Factory, REST API, Azure Function, or Azure SDK)

What specific error or issue are you seeing related to duplicate columns? (Any error messages, unexpected behavior, or screenshots?)

Are you using Merge, Replace, or InsertOrMerge operations during the update?

Which method are you using for authentication—Shared Key, SAS Token, or Azure AD?

Are you trying to update only the audit columns (RefreshType and LastModified), or all columns together?

Is this issue happening during the full load or in a post-load update step in ADF?
Anonymous

2025-06-05T08:18:42.0533333+00:00

@Sri Madhu Just checking in to see if the above answer provided by @ Alex Burlachenko helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

1 answer

Your answer

Anonymous

2025-06-04T11:07:33.7933333+00:00

@Sri Madhu
If you're using ADF for the full load, add a Web Activity at the end of your pipeline that:

Calls an Azure Function or REST API endpoint.

Updates the RefreshType to "Incremental" and updates the LastModified timestamp.

To assist more on this issue please help me with below questions

How are you currently retrieving and updating entities in Azure Table Storage? (e.g., Azure Data Factory, REST API, Azure Function, or Azure SDK)

What specific error or issue are you seeing related to duplicate columns? (Any error messages, unexpected behavior, or screenshots?)

Are you using Merge, Replace, or InsertOrMerge operations during the update?

Which method are you using for authentication—Shared Key, SAS Token, or Azure AD?

Are you trying to update only the audit columns (RefreshType and LastModified), or all columns together?

Is this issue happening during the full load or in a post-load update step in ADF?
Anonymous

2025-06-05T08:18:42.0533333+00:00

@Sri Madhu Just checking in to see if the above answer provided by @ Alex Burlachenko helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

hi there )) thanks for dropping this question on the portal, its super relevant for anyone working with azure tables and adf.

so u got these audit columns in ur azure table right. refresh type and last modified. u want to update them after a full load finishes in adf. U gotta understand how azure table updates work. When u update an entity, u need to include all the properties, not just the ones u wanna change. otherwise, the missing ones will get wiped out. Now about that duplicate columns issue u mentioned. this usually happens when u try to update an entity but u dont include the original values for all properties. adf might treat it like a new set of columns instead of an update.

before u do the update, fetch the current entity from the table. this gives u all the existing values. then, modify only the fields u wanna change (refresh type to incremental, last modified to current timestamp). send the entire entity back with the updates. no missing fields, no duplicates ))

if ur using a web activity or a stored proc in adf, bw sure the payload includes everything. u can also use the azure table storage dataset with upsert, but watch out for those properties. an good example of table operations in adf.

if ur still seeing duplicates, check if u accidentally mapped the same column twice in ur sink dataset. happens to the best of us :)) hope this clears things up.

Best regards,

Alex

and "yes" if you would follow me at Q&A - personaly thx.
P.S. If my answer help to you, please Accept my answer
PPS That is my Answer and not a Comment

https://ctrlaltdel.blog/

Share via

Azure Table Writes and Updates

1 answer

Your answer