Interative Write Operation vs Write Operation

grajee 346 Reputation points
2022-01-30T15:21:26.97+00:00

All,

Why is "Iterative Write" operations which is based on meta-data more expensive than Write operations for Hierarchical Namespace?

Thanks,
Gopi

169597-image.png
169682-image.png

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,365 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee
    2022-02-01T19:40:18.4+00:00

    Hello @grajee ,

    Thanks for the question and using MS Q&A platform.

    My understanding is that you would want to know the difference between iterative write operations compared to that of write operations and how are they billed. If so, please refer to below info which would give you more insights of each functionality and their billing.

    1. Write operations: The following are considered write operations: AppendFile, CreateFilesystem, CreatePath, CreatePathFile, FlushFile, SetFileProperties, SetFilesystemProperties, RenameFile, RenamePathFile, CopyFile

    **How are they billed: **
    When uploading or appending data to existing files, or when reading data from a file, the operation gets chunked into 4MB pieces. You will be billed for each 4MB chunk.

    The Copy File, Rename, Set Properties, etc. would not be charged using the “per 4 MB” rule. They would be charged as a single transaction.

    For files < 4MB, a full transaction will be charged for each file.

    170313-image.png

    1. Iterative write operations : Iterative write operations are operations performed on a folder that requires the system to iterate through all the subfolders and files in that folder to complete. Examples: RenameDirectory, RenamePath, RenamePathDir
      A hierarchical namespace allows operations like folder renames and deletes to be performed in a single atomic operation, which with a flat namespace requires a number of operations proportionate to the number of objects in the structure.

    **How are they billed: **
    For example, if you would want to rename a directory containing 5k files. The rename would be charged as a single metadata operation (iterative writes).

    170263-image.png

    For more info, please refer to below articles:

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators