is multipart upload for large file supported in BlockBlob storage ?

asked 2020-04-29T17:14:17.667+00:00
virender kaushik 1 Reputation point

i have large file of 250GB to upload from my own premises HDFS to azure block blob storage using
distcp command, i am using below command

hadoop distcp \
-Dmapreduce.map.log.level="DEBUG" \
-Dfs.azure.account.key.storageAccount.blob.core.windows.net=<account_key> \
-Dmapreduce.task.timeout=0 -overwrite -i -numListstatusThreads=1 \
/user/test/250gb_file \
wasbs://contianer@storageAccount.blob.core.windows.net/testDir/

Firstly, i am not able to upload file more than size of 195GB. how can we upload the file of size more than 195Gb using distcp command

Secondly, file is large in size so how can we upload file in parallel because sequentially its taking long
time to upload. how can we support multipart in this distcp while uploading file in azure block blob

Not Monitored
Not Monitored
Tag not monitored by Microsoft.
23,655 questions
No comments
{count} votes

1 answer

Sort by: Most helpful
  1. answered 2020-04-29T17:15:31.45+00:00
    Leon Laude 85,026 Reputation points

    Hi,

    Q&A currently supports the products listed in right-hand pane (more to be added later on).

    You can reach the experts in the dedicated Azure Storage forum over here:
    [https://social.msdn.microsoft.com/Forums/azure/en-US/home?forum=windowsazuredata][1]

    (please don't forget to mark helpful replies as answer)

    Best regards,
    Leon

    No comments