question

ChrisAhern-3101 avatar image
0 Votes"
ChrisAhern-3101 asked srbose-msft edited

How to improve (very poor) file read performance with C# Core on an Azure Kubernetes PVC

After having done a number of tests, I've narrowed the performance problem down to a combination of C# and PVC.

Problem: Read a large(ish) file (140MB) using C# Core from a PVC. Here's the (simple) test program:

 using System;
 using System.IO;
    
 namespace cmaApp
 {
     class Program
     {
             private static void readStream(String filePath) {
                 Console.WriteLine($"Reading {filePath}");
                 using (var sr = new StreamReader(new BufferedStream(new FileStream(filePath, FileMode.Open))))
                 {
                         int len = sr.ReadToEnd().Length;
                         Console.WriteLine($"dat len={len}");
                 }
             }
    
             private static void copyFile(String filePath) {
                     File.Copy(filePath, "./t2.dat");
             }
    
         static void Main(string[] args)
         {
             Console.WriteLine("Start!");
             String filePath = "/uploads/3864.dat";    // PVC file
             //String filePath = "./3864.dat";         // Local file
    
             readStream(filePath);
             //copyFile(filePath);
    
             Console.WriteLine("Done!");
         }
     }
 }


Here's what I did, on the command line in the pod:

Test 1. Read the file with C# directly from the PVC (takes 2 mins!!):

 /cmaApp# time dotnet run
 Start!
 Reading /uploads/3864.dat
 dat len=153508096
 Done!
    
 real    2m2.172s
 user    0m3.437s
 sys     0m3.357s

Test 2. Copy the file to a local folder in the pod and read with C# (take 3 seconds):

 cmaApp# time dotnet run
 Start!
 Reading ./3864.dat
 dat len=153508096
 Done!
    
 real    0m2.765s
 user    0m2.910s
 sys     0m0.762s


Test 3. Read the file with cat directly from the PVC (takes 4 secs):

 cmaApp# time cat /uploads/3864.dat |wc
  144187 5543913 153508096
    
 real    0m4.615s
 user    0m0.755s
 sys     0m0.293s

Test 4. Copy the file to a local folder using cp (takes 4 secs):

 cmaApp# time cp /uploads/3864.dat .
    
 real    0m4.358s
 user    0m0.000s
 sys     0m0.213s

Test 5. Copy the file to a local folder using C# File.Copy (takes 13 secs):

 cmaApp# time dotnet run
 Start!
 Done!
    
 real    0m13.042s
 user    0m2.687s
 sys     0m0.913s


Seeing that the linux level cp copies the file fast, it's not a throttling / network issue.

Why is C# Core so slow? What can be done to improve it?

dotnet-csharpazure-kubernetes-serviceazure-filesdotnet-cli
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@ChrisAhern-3101 , thank you for your question. The tests you performed indeed help narrow down the scope of the problem. So what I think it boils down to is "why is the C# code slow to read data from a network storage volume and why is the C# File.Copy method slow to copy a file to a local folder in the pod's filesystem?"

It might be worth knowing what base image you are using with the container, the Linux kernel version of the agent node, node image version and kubernetes version of the AKS cluster/node pools.

You can use the following commands to find this information:

  • Container image:

     kubectl describe pod <podname> -n <namespace> | grep -i image
    

  • Linux kernel version:

  • Node image version:

     az aks nodepool show \
     --resource-group <ResourceGroupName> \
     --cluster-name <AKSClusterName> \
     --name <NodepoolName> \
     --query nodeImageVersion
    

  • Kubernetes version:

    • AKS cluster:

       az aks show -n <ClusterName> -g <ResourceGroupName> --query kubernetesVersion -o tsv
      

    • Node Pool:

       az aks nodepool show --cluster-name <ClusterName> -g <ResourceGroupName> -n <NodepoolName> --query orchestratorVersion -o tsv
      

I have added the dotnet-csharp tag to the question so that a C# expert can contribute to this thread.

0 Votes 0 ·

0 Answers