Copy file from AZure blob to S3 using java code

Question

Hi Can anyone suggest how we can copy files from a blob to AWS S3 location.

I am able to list all the list inside the Blob from Azure but need to send them to S3 location now using java code.
I guess I need to first need to read them from the Azure location and send them to AWS S3 via Using S3 putObject object.
I can upload a local file to AWS S3 using S3 putObject so i am good with AWS but not sure how i can read or get file from blob .

Code i am using to list all Blob files

public void listAllInputStreamBlobbs(String containerName) throws Exception {

    List<String> blobsList = new ArrayList<>();
    CloudBlockBlob blockBlob = null;
    final CloudBlobClient blobClient = getCloudBlobClient(getConnectionString(containerName));
    CloudBlobContainer cloudBlobContainer = null;
    try {
        cloudBlobContainer = getContainerForFolder(blobClient, FS_CLOUD_STORAGE_CONTAINER);
    } catch (Exception e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
    AWSS3BucketServices s3BucketServices = new AWSS3BucketServices();
    // Loop over blobs within the container and output the URI to each of them.
    if (cloudBlobContainer != null) {
        for (ListBlobItem blobItem : cloudBlobContainer.listBlobs("master/", true)) {
            try {
                blockBlob = cloudBlobContainer.getBlockBlobReference(blobItem.getUri().toString());
                String blobnamesVal = blockBlob.getName();
                if (blobnamesVal.contains("/outgoing/")) {
                    // Set only files not incoming/ key as well
                    blobsList.add(blobnamesVal);
                    LOG.info("Print Blob name/url--->" + blobnamesVal);
                    String text = null;
                    if (blobItem instanceof CloudBlob) {
                        // Download the item and save it to a file with the same name.
                        CloudBlob blob = (CloudBlob) blobItem;

                        InputStream dataStream = null;
                        long dataLength = 0;
                        if (blob.exists()) {
                            dataLength = IOUtils.toByteArray(dataStream).length;
                            LOG.info("Size of Data Stream in zie file" + dataLength);

                            byte[] storeds = IOUtils.toByteArray(dataStream);

                            Long size = Long.valueOf(storeds.length);
                            LOG.info("contentBytes.length--->" + storeds);

                            ObjectMetadata objectMetadata = new ObjectMetadata();
                             if (dataStream.available() != 0) {
                                String mime = URLConnection.guessContentTypeFromStream(dataStream);
                                objectMetadata.setContentType(mime);
                                objectMetadata.setContentLength(dataLength);
                                try {
                                    //sending to S3
                                s3BucketServices.uploadInPutStreamToS3Bucket(S3_BUCKET_NAME,"TestBlobtoS3.csv", dataStream, objectMetadata);
                                LOG.info("File uploaded to AWS S 462 line ");

                                } catch (Exception e) {
                                    LOG.info("Error in reading file", e);
                                }
                            }
                        }

                    }
                }

            } catch (URISyntaxException | StorageException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();

            }
        }
    }
}

}

any suggestion or code reference would be appreciated

Answer 1

@Rohit Raj
Here are a few suggestions on how you might copy data from Blob storage to S3. Please check to see if any of these work for your use-case:

https://aws.amazon.com/blogs/storage/one-way-to-migrate-data-from-azure-blob-storage-to-amazon-s3/
https://stackoverflow.com/questions/52989698/copy-data-from-azure-blob-storage-to-aws-s3
https://stackoverflow.com/questions/62711546/how-to-stream-azure-blob-to-aws-s3

Hope this help! Let us know if you have further questions or issues and we will do our best to assist.

-------------------------------

Please don’t forget to "Accept the answer" and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.