How to move data from one table to another table within same workspace of LOG ANALYTICS using Azure Data Factory ?

Question

How to move data from one table to another table within same workspace of LOG ANALYTICS using Azure Data Factory ?

Rishabh Srivastava 60 Microsoft Employee

How to move data

from one table to another table within same workspace of LOG ANALYTICS using Azure Data Factory

2.from one table in log analytics workspace 1 to another table in log analytics 2 workspace using Azure Data Factory ?

Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2024-03-15T06:39:35.7766667+00:00

@Rishabh Srivastava Following up to see if the below answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Rishabh Srivastava 60 Reputation points Microsoft Employee

2024-05-03T01:38:09.7566667+00:00

I followed the suggestion to copy from source table to destination table in the same log analytics workspace. Using Azure Data Factory.

My configuration: Source Linked REST Service to read data from Log Analytics workspace w1 table t1 Destination Linked REST service to write to log analytics workspace w1 table t2

I am using log ingestion API to write to w1 table t2.

Now the issue is how do I map hierarchical data from source to destination as ADF doesn't support this? How can I do mapping?

Accepted answer

0 additional answers

Your answer

Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2024-03-15T06:39:35.7766667+00:00

@Rishabh Srivastava Following up to see if the below answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Rishabh Srivastava 60 Reputation points Microsoft Employee

2024-05-03T01:38:09.7566667+00:00

I followed the suggestion to copy from source table to destination table in the same log analytics workspace. Using Azure Data Factory.

My configuration: Source Linked REST Service to read data from Log Analytics workspace w1 table t1 Destination Linked REST service to write to log analytics workspace w1 table t2

I am using log ingestion API to write to w1 table t2.

Now the issue is how do I map hierarchical data from source to destination as ADF doesn't support this? How can I do mapping?

Answer 1

Smaran Thoomu 24,110 Microsoft External Staff Moderator

Hi @Rishabh Srivastava

Welcome to Microsoft Q&A forum and thanks for reaching out here.

Azure Data Factory (ADF) does not have a direct connector to interact with data stored within Azure Log Analytics workspaces.

One possible option to retrieve data from Log Analytics tables in Azure Data Factory (ADF) is to use the REST connector in ADF to call the Azure Log Analytics REST APIs.

Here is an article by a community volunteer which has detailed steps on how you can get data from Log analytics tables using ADF.

Article: Retrieving Log Analytics Data with Data Factory

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Rishabh Srivastava 60 Reputation points Microsoft Employee

2024-03-17T19:56:00.4333333+00:00
Hi @Smaran Thoomu

SO my use case is we have application logs being sent to log analytics(LA) workspace now we want to copy this log into another log analytics workspace with higher retention period.

Is there an efficient way to

copy data from one LA workspace to other LA workspace Or,

Copy data from one table to other in same LA workspace
Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2024-03-18T06:27:51.7066667+00:00
Hi Rishabh Srivastava

Yes, there are a few ways to copy data from one Log Analytics workspace to another Log Analytics workspace or from one table to another table within the same Log Analytics workspace.

Using Azure Data Factory: You can use Azure Data Factory to copy data from one Log Analytics workspace to another Log Analytics workspace or from one table to another table within the same Log Analytics workspace. You can use the REST API connector in Azure Data Factory to extract data from the source table and then use the REST API connector to insert the data into the sink table. You can also use the Azure Log Analytics connector in Azure Data Factory to extract data from the source table and then use the Azure Log Analytics connector to insert the data into the sink table.

Using Azure Automation: You can use Azure Automation to copy data from one Log Analytics workspace to another Log Analytics workspace or from one table to another table within the same Log Analytics workspace. You can use the Azure Log Analytics PowerShell module to extract data from the source table and then use the same module to insert the data into the sink table.

Using Azure Functions: You can use Azure Functions to copy data from one Log Analytics workspace to another Log Analytics workspace or from one table to another table within the same Log Analytics workspace. You can use the Azure Log Analytics REST API to extract data from the source table and then use the same REST API to insert the data into the sink table.

Using Azure Logic Apps: You can use Azure Logic Apps to copy data from one Log Analytics workspace to another Log Analytics workspace or from one table to another table within the same Log Analytics workspace. You can use the Azure Log Analytics connector in Azure Logic Apps to extract data from the source table and then use the same connector to insert the data into the sink table.

In all of these approaches, you will need to authenticate with Azure AD and obtain an access token to call the Log Analytics REST API. You can use the Azure AD authentication method in the REST API connector or PowerShell module to obtain the access token.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Rishabh Srivastava 60 Reputation points Microsoft Employee

2024-03-18T18:17:58.7+00:00

Thats great. Thanks!
Is there a tutorial guide or some reference to try out POC?
Rishabh Srivastava 60 Reputation points Microsoft Employee

2024-03-18T23:14:18.87+00:00

I couldn't find Azure Log Analytics connector in Azure Data Factory. Is there any wiki ?
Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2024-03-19T05:22:10.6033333+00:00
Hi Rishabh Srivastava

Regarding the Azure Log Analytics connector in Azure Data Factory, it is not a built-in connector. Instead, you can utilize the REST API connector within Azure Data Factory to extract data from Log Analytics tables. Below are the official documentation resources you can refer to.

https://learn.microsoft.com/en-us/azure/azure-monitor/logs/data-platform-logs

https://learn.microsoft.com/en-us/azure/data-factory/monitor-using-azure-monitor

https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/rest-api-walkthrough?tabs=portal#authenticating-azure-monitor-requests

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Rishabh Srivastava 60 Reputation points Microsoft Employee

2024-04-01T19:26:14.48+00:00

Can we use managed identity to manage these resources?
Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2024-04-02T03:50:03.11+00:00

Rishabh Srivastava The scope of your question is different compared to the original scope of the question asked.

I would recommend creating a new thread on the same forum with as much details about your issue as possible. That would make sure that your issue has better visibility in the community.

Share via

How to move data from one table to another table within same workspace of LOG ANALYTICS using Azure Data Factory ?

0 additional answers

Your answer