What should be the optimum block size when trasnfering data from On-prem to ADLS data transfer.

Question

We are using Informatica an on-prem ETL tool to trasnfer data from On-prem server to ADLS blob storage.
While running mappings we are getting Java.netsockettimeout : timeout error. I got some inputs from my team that it is because we dint set the correct ADLS block size. But I am not sure which block size I should set as default so it can be used for various size files.

Answer 1

@S Sharanyu Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

Specify the block size in MB used to write data to ADLS Gen2. Learn more about Block Blobs.
Allowed value is between 4 MB and 100 MB.
By default, ADF automatically determines the block size based on your source store type and data. For non-binary copy into ADLS Gen2, the default block size is 100 MB so as to fit in at most 4.95-TB data. It may be not optimal when your data is not large, especially when you use Self-hosted Integration Runtime with poor network resulting in operation timeout or performance issue. You can explicitly specify a block size, while ensure blockSizeInMB*50000 is big enough to store the data, otherwise copy activity run will fail.

You can use Azcopy, ADF tools and more : Choose an Azure solution for data transfer

Hope this helps!

Kindly let us know if the above helps or you need further assistance on this issue.

---------------------------------------------------------------------------------------------------------------------------------------------------------------

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.