convert a result of collect_list into json using spark with scala

vijendra singh 1

please find the sample below -

after using below code--
val df4 = df3.groupBy("shop_id").agg(collect_list(map($"variant_id",$"variants1")) as ("variants"))

and got data like --

{"shop_id":"1","variants":[{"sku3":{"dual_price_id":-9999,"result":1}},{"sku2":{"dual_price_id":-9999,"result":1}}]}
{"shop_id":"1","variants":[{"sku10":{"dual_price_id":-9999,"result":1}},{"sku3":{"dual_price_id":-9999,"result":1}}]}

but i want data to look like --

{"shop_id":"1","variants":{"sku3":{"dual_price_id":-9999,"result":1}},{"sku2":{"dual_price_id":-9999,"result":1}}}
{"shop_id":"1","variants":{"sku10":{"dual_price_id":-9999,"result":1}},{"sku3":{"dual_price_id":-9999,"result":1}}}

Viorel 112.1K Reputation points

2022-11-11T16:20:42.18+00:00

I think that the required sample is not a JSON.

Maybe you wanted this:

{"shop_id":"1","variants":{"sku3":{"dual_price_id":-9999,"result":1},"sku2":{"dual_price_id":-9999,"result":1}}}.
vijendra singh 1 Reputation point

2022-11-12T00:51:43.963+00:00

@Viorel
yes, exactly this is how i want-

this is how my each rows of data look like-
vijendra singh 1 Reputation point

2022-11-12T00:52:35.043+00:00

@Viorel
yes, exactly this is how i want-

this is how my each rows of data look like-

{"shop_id":"1","mng_number":"00000","item_id":1000,"variants":[{"sku10":{"dual_price_id":-9999,"result":1,"event_type":"DP99: other","regular_price":-9999}},{"sku3":{"dual_price_id":-9999,"result":1,"event_type":"DP99: other","regular_price":-9999}},{"sku2":{"dual_price_id":-9999,"result":1,"event_type":"DP99: other","regular_price":-9999}},{"sku1":{"dual_price_id":-9999,"result":1,"event_type":"DP99: other","regular_price":-9999}},{"sku12":{"dual_price_id":-9999,"result":1,"event_type":"DP99: other","regular_price":-9999}},{"sku11":{"dual_price_id":-9999,"result":1,"event_type":"DP99: other","regular_price":-9999}},{"sku7":{"dual_price_id":-9999,"result":1,"event_type":"DP99: other","regular_price":-9999}},{"sku6":{"dual_price_id":-9999,"result":1,"event_type":"DP99: other","regular_price":-9999}},{"sku5":{"dual_price_id":-9999,"result":1,"event_type":"DP99: other","regular_price":-9999}},{"sku4":{"dual_price_id":-9999,"result":1,"event_type":"DP99: other","regular_price":-9999}},{"sku9":{"dual_price_id":-9999,"result":1,"event_type":"DP99: other","regular_price":-9999}},{"sku8":{"dual_price_id":-9999,"result":1,"event_type":"DP99: other","regular_price":-9999}}]}