Share via

azure-files: high latency / low IOPS when writing to storage v2

tdrabold 1 Reputation point
2021-12-14T15:04:05.197+00:00

Hello! ;-)

I have a question regarding high latency / low IOPS.

In Azure running kubernetes 1.21.2 we (company I work in) have a node size B2s and an StorageV2 (standard SSD, performance tier: hot, replication: LRS).
It hosts a blazor-application which writes the data our editors enter to the before mentioned persistant storageV2 (azure-files) and we run into the problem that the write process takes too long.
We meassured low IOPS (read: 250, write 250, readwrite 150/100).

What can we do to speed up the write process to the storage?

best greetings
Tobias

Azure Files
Azure Files
An Azure service that offers file shares in the cloud.
1,186 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,764 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Sumarigo-MSFT 44,081 Reputation points Microsoft Employee
    2021-12-17T13:55:49.69+00:00

    @tdrabold welcome to Microsoft Q&A Forum, Thank you for posting your query here!

    Please refer to the troubleshooting steps : High latency, low throughput, and general performance issues

    Azure Speed test tool: https://www.azurespeed.com/Azure/Latency

    Here's a guide for troubleshooting high E2E latency: https://learn.microsoft.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting#metrics-show-high-AverageE2ELatency-and-low-AverageServerLatency

    Can you download PerfInsights tool, for Performance data collection Windows
    Using PerfInsights:

    • Execute PerfInsights using the Azure Files scenario: PerfInsights.exe /r azurefiles /sr 9971**9
    • Select Y to agree to share diagnostic information
    • Select Y to agree PerfInsights EULA.
    • Reproduce the performance issue or make sure that it's occurring.
    • When enough data is collected , press any key to stop diagnostic data collection.
    • Retrieve the compressed file created under the current folder with format similar to "CollectedData_YYYY-MM-DD_HH-MM-FFF.zip"

    You might see slow performance when you try to transfer files to the Azure File service.

    If you don't have a specific minimum I/O size requirement, we recommend that you use 1 MiB as the I/O size for optimal performance.

    If you know the final size of a file that you are extending with writes, and your software doesn't have compatibility problems when the unwritten tail on the file contains zeros, then set the file size in advance instead of making every write an extending write.

    This article lists some common problems related to Azure file shares. It provides potential causes and workarounds for when you encounter these problems.

    For testing purpose can you use Azcopy tools and check the speed and let me know the status. For transferring Files I would recommend to use AzCopy and File Storage for any transfer files(upload and download). In Azcopy throughput from a single instance will scale at least 20Gbps

    Additional information: Sometimes your proxies/VMs or whatever runs in the same datacenter, network requests does not leave the local network and the speed then depends on the infrastructure. (routers, firewalls, cables, etc). I'm sure they won't limit speed between their own infrastructure so services hosted on the same network works at full speed.

    If the issue still persists, Please let me know I would like to work closer on this issue

    Please let us know if you have any further queries. I’m happy to assist you further.

    ----------

    Please do not forget to 158545-screenshot-2021-12-10-121802.png and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

  2. tdrabold 1 Reputation point
    2022-02-21T10:25:34.783+00:00

    @Sumarigo-MSFT

    I refered to the troubleshooting steps and found nothing unusual.
    I messured the latancy with Azure Speed test tool and have 61ms up to 221ms.
    I set up a pod running alpine linux und mounted one of our azure files pvc there and ran PerfInsigts there, here is the log:
    176320-perfinsights.log
    176401-perfdiagjs.txt
    I can't see any unusual.

    I hope you can still help me with this.
    I try to explain our scenario again.
    We have a pod running our website with a CMS written by our developers (dotnet application) where our editors can write their articles an publish them. The articles are stored in .json files on Persistant Volume (azure files, StorageV2, Standard LRS) and this write process is too slow, f.e. when 2 editors or more write their articles.
    I messured the IOPS which are at 100 for read and for write with following fio line
    fio -bs=4k -ioengine=libaio -iodepth=1 -direct=1 -rw=randrw -time_based -runtime=30 -refill_buffers -norandommap -randrepeat=0 -group_reporting -name=fio-randread-lat --size=1M -filename=fiotestfile
    Shouldn't the IOPS be at 500??

    0 comments