Share via

How to get rid of the outer array "[" and "]" in the result file after a CSV to JSON pipeline in ADF?

Lynton Grice 0 Reputation points
2023-09-01T23:55:38.1433333+00:00

Hi there,

Hope someone can help me on such a frustrating thing I am having on ADF where I have followed the blog below 100% (I hope) , yet the result JSON file keeps getting an outer array tag as the first "[" and last character "]" --- see first image below to illustrate the array element I want to remove entirely.

https://techcommunity.microsoft.com/t5/fasttrack-for-azure/how-to-convert-csv-file-into-array-of-jsons-in-adf/ba-p/3574859

User's image

So all I need to know is how can I easily remove the first "[" and last "]" from the json stream as looking at the link above, showing this scenario they did not seem to get the "outer array" elements so I am very confused ;-(

My structure / pipeline is similar to what the link above says (with the "dummy" column for grouping later and an "array" column called "records"

User's image

I have done the group by and aggregate exactly as the blog above says:

User's image

User's image

And the SELECT below does not include the "dummy" column just like the post suggested:

User's image

And the sink is just sending through the JSON

User's image

And after running the pipeline the outer array element never seems to go away. I have tried some array functions and even to "stringify" the data and then do something like they describe below, but still no luck

i.e. substringIndex(substringIndex(records, '[', -2),']',2)

https://learn.microsoft.com/en-us/answers/questions/1020768/adf-how-to-remove-the-square-bracket-from-json-as

Can someone please guide me with the best / easiest way to do a simple removal of the first "[" and last "]" in the result JSON file?

I know one could get into some Python code / notebook and do a json.dumps() etc but in this instance would prefer to keep it a simple pipeline like above.

Thanks so much for the help

Lynton

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.


1 answer

Sort by: Most helpful
  1. ShaikMaheer-MSFT 38,636 Reputation points Microsoft Employee Moderator
    2023-09-04T11:09:55.5133333+00:00

    Hi Lynton Grice,

    Thank you for posting query in Microsoft Q&A Platform.

    Yes, you are right. After running dataflow in sink file we will not see it. Glad to know that your issue resolved.

    Kindly consider marking it as accepted answer. Accepted answers help community as well. Thank you.

    Was this answer helpful?


Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.