Muokkaa

Jaa

azcopy sync

  • Artikkeli

Replicates the source location to the destination location. This article provides a detailed reference for the azcopy sync command. To learn more about synchronizing blobs between source and destination locations, see Synchronize with Azure Blob storage by using AzCopy v10. For Azure Files, see Synchronize files.

Note

AzCopy does not support scenarios where the source or destination is being actively changed during the transfer.

Synopsis

The last modified times are used for comparison. The file is skipped if the last modified time in the destination is more recent. Alternatively, you can use the --compare-hash flag to transfer only files which differ in their MD5 hash. The supported pairs are:

  • Local <-> Azure Blob / Azure File (either SAS or OAuth authentication can be used)
  • Azure Blob <-> Azure Blob (either SAS or OAuth authentication can be used)
  • Azure File <-> Azure File (Source must include a SAS or is publicly accessible; SAS authentication should be used for destination)
  • Azure Blob <-> Azure File

The sync command differs from the copy command in several ways:

  1. By default, the recursive flag is true and sync copies all subdirectories. Sync copies only the top-level files inside a directory if the recursive flag is false.
  2. When syncing between virtual directories, add a trailing slash to the path (refer to examples) if there's a blob with the same name as one of the virtual directories.
  3. If the 'delete-destination' flag is set to true or prompt, then sync will delete files and blobs at the destination that aren't present at the source.

Guidelines

By default, the sync command compares file names and last modified timestamps. You can override that behavior to use MD5 hashes instead of last modified timestamps by using the --compare-hash flag. Set the --delete-destination optional flag to a value of true or prompt to delete files in the destination directory if those files no longer exist in the source directory.

  • If you set the --delete-destination flag to true, AzCopy deletes files without providing a prompt. If you want a prompt to appear before AzCopy deletes a file, set the --delete-destination flag to prompt.

  • If you plan to set the --delete-destination flag to prompt or false, consider using the copy command instead of the sync command and set the --overwrite parameter to ifSourceNewer. The copy command consumes less memory and incurs less billing costs because a copy operation doesn't have to index the source or destination prior to moving files.

  • If you don't plan to use the --compare-hash flag, then the machine on which you run the sync command should have an accurate system clock because the last modified times are critical in determining whether a file should be transferred. If your system has significant clock skew, avoid modifying files at the destination too close to the time that you plan to run a sync command.

  • AzCopy uses server-to-server APIs to synchronize data between storage accounts. That means that data is copied directly between storage servers. However, AzCopy does set up and monitor each transfer, and for larger storage accounts (For example, accounts that contain millions of blobs), AzCopy might require a substantial amount of compute resources to accomplish these tasks. Therefore, if you are running AzCopy from Virtual Machine (VM), make sure that the VM has enough cores/memory to handle the load.

  • For Blob Storage, you can prevent accidental deletions by making sure to enable the soft delete feature before you use the --delete-destination=prompt|true flag.

Advanced

Note that if you don't specify a file extension, AzCopy automatically detects the content type of the files when uploading from the local disk, based on the file extension or content.

The built-in lookup table is small but on Unix it's augmented by the local system's mime.types file(s) if available under one or more of these names:

  • /etc/mime.types
  • /etc/apache2/mime.types
  • /etc/apache/mime.types

On Windows, MIME types are extracted from the registry.

By default sync works off of the last modified times unless you override that default behavior by using the --compare-hash flag. So in the case of Azure File <-> Azure File, the header field Last-Modified is used instead of x-ms-file-change-time, which means that metadata changes at the source can also trigger a full copy.

azcopy sync [flags]

Examples

Sync a single file:

azcopy sync "/path/to/file.txt" "https://[account].blob.core.windows.net/[container]/[path/to/blob]"

Same as above, but also compute an MD5 hash of the file content, and then save that MD5 hash as the blob's Content-MD5 property.

azcopy sync "/path/to/file.txt" "https://[account].blob.core.windows.net/[container]/[path/to/blob]" --put-md5

Sync an entire directory including its subdirectories (note that recursive is by default on):

azcopy sync "/path/to/dir" "https://[account].blob.core.windows.net/[container]/[path/to/virtual/dir]" or azcopy sync "/path/to/dir" "https://[account].blob.core.windows.net/[container]/[path/to/virtual/dir]" --put-md5

Sync only the files inside of a directory but not subdirectories or the files inside of subdirectories:

azcopy sync "/path/to/dir" "https://[account].blob.core.windows.net/[container]/[path/to/virtual/dir]" --recursive=false

Sync a subset of files in a directory (For example: only jpg and pdf files, or if the file name is "exactName"):

azcopy sync "/path/to/dir" "https://[account].blob.core.windows.net/[container]/[path/to/virtual/dir]" --include-pattern="*.jpg;*.pdf;exactName"

Sync an entire directory but exclude certain files from the scope (For example: every file that starts with foo or ends with bar):

azcopy sync "/path/to/dir" "https://[account].blob.core.windows.net/[container]/[path/to/virtual/dir]" --exclude-pattern="foo*;*bar"

Sync a single blob:

azcopy sync "https://[account].blob.core.windows.net/[container]/[path/to/blob]?[SAS]" "https://[account].blob.core.windows.net/[container]/[path/to/blob]"

Sync a virtual directory:

azcopy sync "https://[account].blob.core.windows.net/[container]/[path/to/virtual/dir]?[SAS]" "https://[account].blob.core.windows.net/[container]/[path/to/virtual/dir]" --recursive=true

Sync a virtual directory that has the same name as a blob (add a trailing slash to the path in order to disambiguate):

azcopy sync "https://[account].blob.core.windows.net/[container]/[path/to/virtual/dir]/?[SAS]" "https://[account].blob.core.windows.net/[container]/[path/to/virtual/dir]/" --recursive=true

Sync an Azure File directory (same syntax as Blob):

azcopy sync "https://[account].file.core.windows.net/[share]/[path/to/dir]?[SAS]" "https://[account].file.core.windows.net/[share]/[path/to/dir]" --recursive=true

Note: if include and exclude flags are used together, only files matching the include patterns are used, but those matching the exclude patterns are ignored.

Options

--block-size-mb (float) Use this block size (specified in MiB) when uploading to Azure Storage or downloading from Azure Storage. Default is automatically calculated based on file size. Decimal fractions are allowed (For example: 0.25). When uploading or downloading, the maximum allowed block size is 0.75 * AZCOPY_BUFFER_GB. To learn more, see Optimize memory use.

--check-md5 (string) Specifies how strictly MD5 hashes should be validated when downloading. This option is only available when downloading. Available values include: NoCheck, LogOnly, FailIfDifferent, FailIfDifferentOrMissing. (default 'FailIfDifferent'). (default "FailIfDifferent")

--compare-hash (string) Inform sync to rely on hashes as an alternative to Last Modified Time (LMT). Missing hashes at a remote source will throw an error. (None, MD5) Default: None (default "None")

--cpk-by-name (string) Client provided key by name let clients that make requests against Azure Blob storage an option to provide an encryption key on a per-request basis. Provided key name will be fetched from Azure Key Vault and will be used to encrypt the data

--cpk-by-value Client provided key by name let clients that make requests against Azure Blob storage an option to provide an encryption key on a per-request basis. Provided key and its hash will be fetched from environment variables

--delete-destination (string) Defines whether to delete extra files from the destination that aren't present at the source. Could be set to true, false, or prompt. If set to prompt, the user will be asked a question before scheduling files and blobs for deletion. (default 'false'). (default "false")

--dry-run Prints the path of files that would be copied or removed by the sync command. This flag doesn't copy or remove the actual files.

--exclude-attributes (string) (Windows only) Exclude files whose attributes match the attribute list. For example: A;S;R

--exclude-path (string) Exclude these paths when comparing the source against the destination. This option doesn't support wildcard characters (*). Checks relative path prefix(For example: myFolder;myFolder/subDirName/file.pdf).

--exclude-pattern (string) Exclude files where the name matches the pattern list. For example: .jpg;.pdf;exactName

--exclude-regex (string) Exclude the relative path of the files that match with the regular expressions. Separate regular expressions with ';'.

--force-if-read-only When overwriting an existing file on Windows or Azure Files, force the overwrite to work even if the existing file has its read-only attribute set.

--from-to (string) Optionally specifies the source destination combination. For Example: LocalBlob, BlobLocal, LocalFile, FileLocal, BlobFile, FileBlob, etc.

-h, --help help for sync

--include-attributes (string) (Windows only) Include only files whose attributes match the attribute list. For example: A;S;R

--include-pattern (string) Include only files where the name matches the pattern list. For example: .jpg;.pdf;exactName

--include-regex (string) Include the relative path of the files that match with the regular expressions. Separate regular expressions with ';'.

--hash-meta-dir When using --local-hash-storage-mode=HiddenFiles, you can specify an alternate directory to store hash metadata files in (as opposed to next to the related files in the source).

--local-hash-storage-mode Specify an alternative way to cache file hashes. Valid options are: HiddenFiles (OS Agnostic), XAttr (Linux/MacOS only (requires user_xattr on all file systems traversed sat the source), AlternateDataStreams (Windows only. requires named streams on target volume).

--mirror-mode Disable last-modified-time based comparison and overwrites the conflicting files and blobs at the destination if this flag is set to true. Default is false

--put-blob-size-mb Use this size (specified in MiB) as a threshold to determine whether to upload a blob as a single PUT request when uploading to Azure Storage. The default value is automatically calculated based on file size. Decimal fractions are allowed (For example: 0.25).

--preserve-permissions False by default. Preserves ACLs between aware resources (Windows and Azure Files, or ADLS Gen 2 to ADLS Gen 2). For Hierarchical Namespace accounts, you'll need a container SAS or OAuth token with Modify Ownership and Modify Permissions permissions. For downloads, you'll also need the --backup flag to restore permissions where the new Owner won't be the user running AzCopy. This flag applies to both files and folders, unless a file-only filter is specified (for example, include-pattern).

--preserve-posix-properties- False by default. Preserves property information gleaned from stat or statx into object metadata.

--preserve-smb-info For SMB-aware locations, flag will be set to true by default. Preserves SMB property info (last write time, creation time, attribute bits) between SMB-aware resources (Azure Files). This flag applies to both files and folders, unless a file-only filter is specified (for example, include-pattern). The info transferred for folders is the same as that for files, except for Last Write Time that isn't preserved for folders. (default true)

--put-md5 Create an MD5 hash of each file, and save the hash as the Content-MD5 property of the destination blob or file. (By default the hash is NOT created.) Only available when uploading.

--recursive True by default, look into subdirectories recursively when syncing between directories. (default true). (default true)

--s2s-preserve-access-tier Preserve access tier during service to service copy. Refer to Azure Blob storage: hot, cool, and archive access tiers to ensure destination storage account supports setting access tier. In the cases that setting access tier isn't supported, please use s2sPreserveAccessTier=false to bypass copying access tier. (default true). (default true)

--s2s-preserve-blob-tags Preserve index tags during service to service sync from one blob storage to another

--trailing-dot Enabled by default to treat file share related operations in a safe manner. Available options: Enable, Disable. Choose Disable to go back to legacy (potentially unsafe) treatment of trailing dot files where the file service will trim any trailing dots in paths. This can result in potential data corruption if the transfer contains two paths that differ only by a trailing dot (For example mypath and mypath.). If this flag is set to Disable and AzCopy encounters a trailing dot file, it will warn customers in the scanning log but will not attempt to abort the operation. If the destination does not support trailing dot files (Windows or Blob Storage), AzCopy will fail if the trailing dot file is the root of the transfer and skip any trailing dot paths encountered during enumeration.

Options inherited from parent commands

--log-level (string) Define the log verbosity for the log file, available levels: INFO(all requests and responses), WARNING(slow responses), ERROR(only failed requests), and NONE(no output logs). (default INFO). (default "INFO")

--cap-mbps (float) Caps the transfer rate, in megabits per second. Moment-by-moment throughput might vary slightly from the cap. If this option is set to zero, or it's omitted, the throughput isn't capped.

--output-type (string) Format of the command's output. The choices include: text, json. The default value is 'text'. (default "text")

--trusted-microsoft-suffixes (string) Specifies other domain suffixes where Microsoft Entra login tokens may be sent. The default is '.core.windows.net;.core.chinacloudapi.cn;.core.cloudapi.de;.core.usgovcloudapi.net;*.storage.azure.net'. Any listed here are added to the default. For security, you should only put Microsoft Azure domains here. Separate multiple entries with semi-colons.

See also