AdlsClient.BulkUpload Method

Definition

Namespace:

Microsoft.Azure.DataLake.Store

Assembly:

Microsoft.Azure.DataLake.Store.dll

Package:

Microsoft.Azure.DataLake.Store v1.1.24

Overloads

BulkUpload(String, String, Int32, IfExists, IProgress<TransferStatus>, Boolean, Boolean, Boolean, CancellationToken)

Upload directory or file from local to remote. Transfers the contents under source directory under the destination directory. Transfers the source file and saves it as the destination path. This method does not throw any exception for any entry's transfer failure. Refer the return value TransferStatus to get the status/exception of each entry's transfer. It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2. By default files are uploaded at new line boundaries. However if files does not have newline within 4MB chunks, the transfer will fail. In that case it is required to pass true to isBinary to avoid uploads at newline boundaries.

BulkUpload(String, String, Int32, IfExists, Boolean, IProgress<TransferStatus>, Boolean, Boolean, Boolean, CancellationToken)

Upload directory or file from local to remote. Transfers the contents under source directory under the destination directory. Transfers the source file and saves it as the destination path. This method does not throw any exception for any entry's transfer failure. Refer the return value TransferStatus to get the status/exception of each entry's transfer. By default logs the transfer progress in system's temp path, so that user can recover using resume if upload has crashed. This progress logging can be disabled using disableTransferLogging. It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2. By default files are uploaded at new line boundaries. However if files does not have newline within 4MB chunks, the transfer will fail. In that case it is required to pass true to isBinary to avoid uploads at newline boundaries.

BulkUpload(String, String, Int32, IfExists, IProgress<TransferStatus>, Boolean, Boolean, Boolean, CancellationToken)

Upload directory or file from local to remote. Transfers the contents under source directory under the destination directory. Transfers the source file and saves it as the destination path. This method does not throw any exception for any entry's transfer failure. Refer the return value TransferStatus to get the status/exception of each entry's transfer. It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2. By default files are uploaded at new line boundaries. However if files does not have newline within 4MB chunks, the transfer will fail. In that case it is required to pass true to isBinary to avoid uploads at newline boundaries.

public virtual Microsoft.Azure.DataLake.Store.FileTransfer.TransferStatus BulkUpload (string srcPath, string destPath, int numThreads = -1, Microsoft.Azure.DataLake.Store.IfExists shouldOverwrite = Microsoft.Azure.DataLake.Store.IfExists.Overwrite, IProgress<Microsoft.Azure.DataLake.Store.FileTransfer.TransferStatus> progressTracker = default, bool notRecurse = false, bool resume = false, bool isBinary = false, System.Threading.CancellationToken cancelToken = default);

abstract member BulkUpload : string * string * int * Microsoft.Azure.DataLake.Store.IfExists * IProgress<Microsoft.Azure.DataLake.Store.FileTransfer.TransferStatus> * bool * bool * bool * System.Threading.CancellationToken -> Microsoft.Azure.DataLake.Store.FileTransfer.TransferStatus
override this.BulkUpload : string * string * int * Microsoft.Azure.DataLake.Store.IfExists * IProgress<Microsoft.Azure.DataLake.Store.FileTransfer.TransferStatus> * bool * bool * bool * System.Threading.CancellationToken -> Microsoft.Azure.DataLake.Store.FileTransfer.TransferStatus

Public Overridable Function BulkUpload (srcPath As String, destPath As String, Optional numThreads As Integer = -1, Optional shouldOverwrite As IfExists = Microsoft.Azure.DataLake.Store.IfExists.Overwrite, Optional progressTracker As IProgress(Of TransferStatus) = Nothing, Optional notRecurse As Boolean = false, Optional resume As Boolean = false, Optional isBinary As Boolean = false, Optional cancelToken As CancellationToken = Nothing) As TransferStatus

Parameters

srcPath

String

Local source path

destPath

String

Remote destination path - It should always be a directory.

numThreads

Int32

Number of threads- if not passed will take default number of threads (8 times the number of physical cores)

shouldOverwrite

IfExists

Whether to overwrite or skip if the destination exists

progressTracker

IProgress<TransferStatus>

Progresstracker to track progress of file transfer

notRecurse

Boolean

If true then does an enumeration till level one else does recursive enumeration

resume

Boolean

If true then we want to resume from last transfer

isBinary

Boolean

If false then writes files to data lake at newline boundaries, however if the file has no newline within 4MB chunks it will throw exception. If true, then upload at new line boundaries is not guranteed but the upload will be faster. By default false, if file has no newlines within 4MB chunks true should be apssed

cancelToken

CancellationToken

Cancellation token

Returns

TransferStatus

Transfer Status encapsulating the details of upload

BulkUpload(String, String, Int32, IfExists, Boolean, IProgress<TransferStatus>, Boolean, Boolean, Boolean, CancellationToken)

Upload directory or file from local to remote. Transfers the contents under source directory under the destination directory. Transfers the source file and saves it as the destination path. This method does not throw any exception for any entry's transfer failure. Refer the return value TransferStatus to get the status/exception of each entry's transfer. By default logs the transfer progress in system's temp path, so that user can recover using resume if upload has crashed. This progress logging can be disabled using disableTransferLogging. It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2. By default files are uploaded at new line boundaries. However if files does not have newline within 4MB chunks, the transfer will fail. In that case it is required to pass true to isBinary to avoid uploads at newline boundaries.

public virtual Microsoft.Azure.DataLake.Store.FileTransfer.TransferStatus BulkUpload (string srcPath, string destPath, int numThreads, Microsoft.Azure.DataLake.Store.IfExists shouldOverwrite, bool disableTransferLogging, IProgress<Microsoft.Azure.DataLake.Store.FileTransfer.TransferStatus> progressTracker, bool notRecurse, bool resume, bool isBinary, System.Threading.CancellationToken cancelToken);

abstract member BulkUpload : string * string * int * Microsoft.Azure.DataLake.Store.IfExists * bool * IProgress<Microsoft.Azure.DataLake.Store.FileTransfer.TransferStatus> * bool * bool * bool * System.Threading.CancellationToken -> Microsoft.Azure.DataLake.Store.FileTransfer.TransferStatus
override this.BulkUpload : string * string * int * Microsoft.Azure.DataLake.Store.IfExists * bool * IProgress<Microsoft.Azure.DataLake.Store.FileTransfer.TransferStatus> * bool * bool * bool * System.Threading.CancellationToken -> Microsoft.Azure.DataLake.Store.FileTransfer.TransferStatus

Public Overridable Function BulkUpload (srcPath As String, destPath As String, numThreads As Integer, shouldOverwrite As IfExists, disableTransferLogging As Boolean, progressTracker As IProgress(Of TransferStatus), notRecurse As Boolean, resume As Boolean, isBinary As Boolean, cancelToken As CancellationToken) As TransferStatus

Parameters

srcPath

String

Local source path

destPath

String

Remote destination path - It should always be a directory.

numThreads

Int32

Number of threads- Default -1, if not passed will take default number of threads (8 times the number of physical cores)

shouldOverwrite

IfExists

Whether to overwrite or skip if the destination exists, Default IfExists.Overwrite

disableTransferLogging

Boolean

If true, logging of transfer progress is disabled. This and resume cannot be true at same time. Default false

progressTracker

IProgress<TransferStatus>

Progresstracker to track progress of file transfer, Default null

notRecurse

Boolean

If true then does an enumeration till level one else does recursive enumeration, Default false

resume

Boolean

If true then we want to resume from last transfer, Default false

isBinary

Boolean

If false then writes files to data lake at newline boundaries, however if the file has no newline within 4MB chunks it will throw exception. If true, then upload at new line boundaries is not guranteed but the upload will be faster. By default false, if file has no newlines within 4MB chunks true should be apssed

cancelToken

CancellationToken

Cancellation token

Returns

TransferStatus

Transfer Status encapsulating the details of upload