changed value of file pattern to set of objects in copy activity under sink but still getting wrong output

Question

changed value of file pattern to set of objects in copy activity under sink but still getting wrong output

azure_learner 20

Hi All,

so i changed the value of property file pattern to set of objects from UI and saved changed and ran the pipeline but still the output file is giving me results in this format which is wrong

existing result

[
	{"empid" : 1, "empname" : "A"},
	{"empid" : 2, "empname" : "B"},
	{"empid" : 3, "empname" : "C"}
]

expected result :

{"empid" : 1, "empname" : "A"},
{"empid" : 2, "empname" : "B"},
{"empid" : 3, "empname" : "C"}

pipeline UI :

file pattern value changed to set of objects from UI

Pipeline backend JSON file pattern value in backend JSON code

can anyone please help with why its not working as expected?

thanks in advance :)

Venkat Reddy Navari 940 Reputation points Microsoft External Staff

2025-04-07T15:31:30.64+00:00
Hi @azure_learner You’ve updated the file pattern in your Copy Activity to Set of objects, but it looks like the output is still coming through as a single JSON array wrapped in square brackets. From what you described, it seems you're expecting each object to appear on a separate line instead of being grouped together in an array.

The following points might provide some clarity and lead to a resolution

Sink Dataset Format: Please ensure that the dataset format is set to JSON and that the File pattern is correctly set to Set of objects—not only in the UI but also in the backend JSON code of the pipeline.

Pipeline Publication: After changing the file pattern, make sure the pipeline was published before running the pipeline again. Sometimes, the UI changes don’t take effect until published.

Mapping/Data Flow: If you're using a Mapping Data Flow, confirm that the structure being written is flattened properly. Even with the right file pattern, nested structures may still produce an array-like format.

Expected Format Clarification: The Set of objects setting is designed to produce JSON Lines (newline-delimited JSON), which means each object appears on a new line without square brackets. If this still isn’t happening, please check that there are no other conflicting sink settings such as compression or encoding.

For more details, you can refer to the official Microsoft documentation: https://learn.microsoft.com/en-us/azure/data-factory/format-json

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
azure_learner 20 Reputation points

2025-04-08T08:06:12.93+00:00

Thanks Venkat Reddy Navari for quick response :)

Accepted answer

0 additional answers

Your answer

Venkat Reddy Navari 940 Reputation points Microsoft External Staff

2025-04-07T15:31:30.64+00:00

Hi @azure_learner You’ve updated the file pattern in your Copy Activity to Set of objects, but it looks like the output is still coming through as a single JSON array wrapped in square brackets. From what you described, it seems you're expecting each object to appear on a separate line instead of being grouped together in an array.

The following points might provide some clarity and lead to a resolution

Sink Dataset Format: Please ensure that the dataset format is set to JSON and that the File pattern is correctly set to Set of objects—not only in the UI but also in the backend JSON code of the pipeline.

Pipeline Publication: After changing the file pattern, make sure the pipeline was published before running the pipeline again. Sometimes, the UI changes don’t take effect until published.

Mapping/Data Flow: If you're using a Mapping Data Flow, confirm that the structure being written is flattened properly. Even with the right file pattern, nested structures may still produce an array-like format.

Expected Format Clarification: The Set of objects setting is designed to produce JSON Lines (newline-delimited JSON), which means each object appears on a new line without square brackets. If this still isn’t happening, please check that there are no other conflicting sink settings such as compression or encoding.

For more details, you can refer to the official Microsoft documentation: https://learn.microsoft.com/en-us/azure/data-factory/format-json

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
azure_learner 20 Reputation points

2025-04-08T08:06:12.93+00:00

Thanks Venkat Reddy Navari for quick response :)

Answer 1

Rakesh Govindula 965 Microsoft External Staff

Hi @azure_learner,

i changed the value of property file pattern to set of objects from UI and saved changed and ran the pipeline but still the output file is giving me results in this format which is wrong

The value is still arrayofObjects in the Pipeline JSON even after changing in the UI suggest that this might be an intermittent issue.

You can try the below trouble shooting steps first from your end:

Publish the pipeline changes and re-open the ADF workspace again.
Clear the browser cache and also try with another browser.

If still facing the issue, instead of changing in the UI, try to edit in the Pipeline JSON itself and save it.


"sink": {

                        "type": "JsonSink",

                        "storeSettings": {

                            "type": "AzureBlobFSWriteSettings"

                        },

                        "formatSettings": {

                            "type": "JsonWriteSettings",

                            "filePattern": "setOfObjects"

                        }

                    },

enter image description here

The commas in between the objects will give the invalid JSON. The correct result JSON from the above pipeline run looks as shown below.

enter image description here

The dataflow will also give the same result file.

Hope this helps.

If the answer is helpful, please click Accept Answer and kindly upvote it. If you have any further questions about this answer, please click Comment.

azure_learner 20 Reputation points

2025-04-08T08:03:55.6233333+00:00
Thanks @Rakesh Govindula

way 1 : since its my child branch i can not publish it but i once again changed file pattern = set of objects and saved and just refreshed the page and then checked and its back to old value of array of objects automatically

way 2 : did same changes as way 1 and saved and then re-opened ADF workspace but same issue as way 1

way 3 : cleared browser cache and did same thing as way 1 and same issue

way 4 : tried in another browser and did same thing as way 1 and same issue way 5 : tried to edit in the JSON and it worked !!

thank a lot for your help @Rakesh Govindula :)

so then its a bug in microsoft side if the user have to explicitly change the value in pipeline JSON. if this is not the right place can you please guide me to right place to post about this bug ? because of this bug i had to spend a lot of time and at the end i thought i never gonna get this output or i will have to spend more time to find out an alternative. this also leads to wrong information getting forwarded to other people if they ask me and i will say set of object does not give this result because i tried and it did not work, so technically am right based on my experience and observation but its actually wrong and its due to the bug of microsoft
Rakesh Govindula 965 Reputation points Microsoft External Staff

2025-04-08T09:20:45.5633333+00:00

Hello @azure_learner,

Glad one of the approaches is working for you for time being. As it is working fine from my end, I think it's the issue from your end and not a bug. Yes, this is the place for your request. Can you please provide the required details in the private messages?

Share via

changed value of file pattern to set of objects in copy activity under sink but still getting wrong output

0 additional answers

Your answer