ADF - JSON processing

Question

Hey all,

A newbie on ADF. I have a JSON file - its an array of objects like below:

[{"Timestamp":"2022-10-26 19:21:23","Category":"Warning","OperationName":"Test1","Message":"Test1'"}

]

The ADF needs to take the file from source and send it to a target location but it needs to do some processing in between.

As i am more of a C# guy , i tried the processing in Visual studio and its working. The equivalent C# code in Visual studio is somethng like this

 string jsonvar = the json message above

 var datestring = DateTime.UtcNow.ToString("r");

var jsonBytes = Encoding.UTF8.GetBytes(jsonvar);

Now I am trying to achieve this in ADF. For the first step, to store the message in a variable jsonvar,

How do i get this entire message as a string variable within ADF?

Also step2,3: are there equivalent functions in ADF ?

Please help.

Answer 1

@Biju Mathew Hello and welcome to Microsoft Q&A.

So I understand you want to do some transformation on your JSON file, but are not sure how to go about it.

For most transformation in Data Factory, you want to use Data Flow. Set up properly, Data Flow's Source transformation will ingest your JSON file, and each element of the outermost array will be treated as a "row". Then using a Derived Column transformation you can alter each property as needed.

If you need this as part of the control logic in the pipeline, then you should use the Lookup Activity followed by Set Variable activity.

For getting and formatting the time in pipeline expressions:

@formatDateTime( utcnow() , "r" )

I'm not sure of the end goal of var jsonBytes = Encoding.UTF8.GetBytes(jsonvar);

If you just want to change encoding, that is set in the Dataset properties.

Let me know the greater context of your ask and I can provide better help.

Answer 2

Thanks for this @MartinJaffer-MSFT

To give you the wider context, we want to import logs into Azure Log Analytics using ADF. we found a article that says you could insert logs into Log Analytics using REST API>

That article is this [https://learn.microsoft.com/en-us/azure/azure-monitor/logs/data-collector-api?tabs=powershell

There is a sample C# code in a section in that article Please have a look here

[https://learn.microsoft.com/en-us/azure/azure-monitor/logs/data-collector-api?tabs=c-sharp

Please look at the C# code sample there.

I am trying to see if somehow this C# code funtionality can be replicated from within ADF. There is encoding, HMAC SHA256 signature computation etc. and i am not sure how we do all this within ADF

Please have a look and any guidance would be very much appreciated.

Thanks in advance

Answer 3

Thanks @MartinJaffer-MSFT for this.

I would like to give you the wider context: we are trying to insert logs into Azure Log Analytics using ADF.

We found an article that says you could use REST API to insert into Log Analytics.

[https://learn.microsoft.com/en-us/azure/azure-monitor/logs/data-collector-api?tabs=c-sharp

If you go to the Sample Requests section within the artilce, you would find a C# code.

I am able to run this code from within a Windows application in Visual Studio. However, we have been asked to see if this code functionality can be replicated within ADF - so ADF will ingest a file and insert into Log Analytics.

If you look at the code, there is this bytearray conversion, HMAC SHA256 signature construction etc invoved in the API invocation.

Is there any way we can replicate this functionality in ADF?

any guidance will be very much appreciated.

Thanks in advance