Azcopy in Databricks

yamuna devi sigireddi 26 Reputation points
2022-05-09T09:16:15.867+00:00

I need to copy data from one storage account to another . How do I use azcopy in Databricks for this? I downloaded azcopy in local but how do I use it with Databricks? Or is there anyway to execute this in Data factory?

Azure Storage Explorer
Azure Storage Explorer
An Azure tool that is used to manage cloud storage resources on Windows, macOS, and Linux.
241 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,078 questions
{count} vote

2 answers

Sort by: Most helpful
  1. Pratik Somaiya 4,206 Reputation points
    2022-05-09T11:12:05.403+00:00

    Hello @yamuna devi sigireddi

    Is there an issue with AzCopy on local as mostly it is used from local system

    If you need to try it from Databricks then you can try referring to this article here

    Let me know of any queries

    0 comments No comments

  2. PRADEEPCHEEKATLA-MSFT 84,936 Reputation points Microsoft Employee
    2022-05-10T04:31:13.65+00:00

    Hello @yamuna devi sigireddi ,

    Thanks for the question and using MS Q&A platform.

    Unfortunately, you cannot use AzCopy in Databricks.

    In Azure Databricks, you can use Databricks Utilities to copy from one mounted storage account to another mounted storage accounts.

    dbutils.fs.help("cp") => cp(from: String, to: String, recurse: boolean = false): boolean -> Copies a file or directory, possibly across FileSystems

    /**  
    * Copies a file or directory, possibly across FileSystems.  
    *  
    * Example: cp("/mnt/my-folder/a", "dbfs://a/b")  
    *  
    * @param from FileSystem URI of the source file or directory  
    * @param to FileSystem URI of the destination file or directory  
    * @param recurse if true, all files and directories will be recursively copied  
    * @return true if all files were successfully copied  
    */  
    cp(from: java.lang.String, to: java.lang.String, recurse: boolean = false): boolean  
    

    Note: AzCopy is a command-line utility that you can use to copy blobs or files to or from a storage account. This article- Get started with AzCopy helps you download AzCopy, connect to your storage account, and then transfer data.

    Yes, you can also use use Azure Data factory to copy from one storage account to another storage account.

    For more details, refer to Copy and transform data in Azure Blob storage by using Azure Data Factory or Azure Synapse Analytics.

    There are a variety of Azure data transfer solutions available for customers: Choose an Azure solution for data transfer.

    200430-image.png

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators