question

sdobbi-6684 avatar image
0 Votes"
sdobbi-6684 asked sdobbi-6684 answered

System.Text.Json.JsonSerializer.Deserialize slowness in Azure Kubernetes Service

In a .NET 6.0 Web API running in Azure Kubernetes Service (using a Docker Image in Azure Container Registry), System.Text.Json.JsonSerializer.Deserialize<T>(jsonToDeserialize, options) is taking 2-3 seconds. Locally on a Windows 10 computer, it takes about .40 seconds. Does anyone know how to solve the slowness with JsonSerializer.Deserialize in Azure Kubernetes Service?

azure-kubernetes-servicedotnet-standard
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

shivapatpi-MSFT avatar image
0 Votes"
shivapatpi-MSFT answered

Hello @sdobbi-6684 ,
We would like to know more details about your problem statement but based upon the available information

From AKS point of I was thinking below suggestions:

1) Validate the worker Nodes VM size which is being used , by default AKS gets deployed with Standard_DS2_v2 - try to have a new nodepool with higher configuration (Like more CPU/Memory)
2) When the Deserializer statement is being executed , in other window check the utilization of the node ( kubectl top nodes (or) kubectl top pods)
3) Consider using Ephemeral OS disks for AKS node pools https://docs.microsoft.com/en-us/samples/azure-samples/aks-ephemeral-os-disk/aks-ephemeral-os-disk/
4) Is this running on Linux/Windows container? If it is a windows container - I would suggest higher VM size for the worker nodes (Just play around with available options while creating the node pool)


Coming to Code point of view:

1) The data which you are trying to serialize where exactly it is getting downloaded? (When deployed in AKS and when running locally)
2) Can you provide a sample piece of code which we can test it from our end ?
3) How the docker file is being generated , is that from VS Code ?


Hope that helps !

Regards,
Shiva

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

sdobbi-6684 avatar image
1 Vote"
sdobbi-6684 answered

Hi @shivapatpi-MSFT ,

Thank you for your detailed response.

The problem was solved in AKS by re-using the same JsonSerializerOptions instance for each JsonSerializer.Serialize and JsonSerializer.Deserialize. I'm not sure why it behaves differently locally running the code in VS 2022 on Windows 10.

The code makes a call to an API and performs JsonSerializer.Deserialize on the data that is returned. The data that is returned is very large.

It is a Linux container.

Seth

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

sdobbi-6684 avatar image
0 Votes"
sdobbi-6684 answered

Hi @shivapatpi-MSFT ,

After I wrote that the problem was solved, the problem came back. Therefore, I restarted the Pod in AKS and ran tests where 106 JsonSerializer.Deserialize operations were performed and all of them were less than 1 second which is good. Any idea why this would be solved by restarting the Pod in AKS? My concern is that this will keep happening and the Pod will need to be restarted.

Thanks,
Seth

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.