Azure Databricks read from S3 - costs

Question

Azure Databricks read from S3 - costs

MH-3502 21

Hi,

i know that one can directly use data stored in an AWS S3-Bucket in an Azure Databricks Notebook. What i wonder is: will there be any additional (apart from the "normal" Databricks cluster costs) costs generated by doing so? I´m thinking about data egress from AWS and/or transfer bandwith etc.

If yes, how could we estimate the amount of data that would be billed? Would only data that is physically "saved" to e.g. an Azure Storage Account after processing it in Databricks be taken in to account, or everything that is originally read into Databricks or....

Thank you!

Accepted answer

0 additional answers

Your answer

Answer 1

PRADEEPCHEEKATLA 90,651 Moderator

Hello @MH-3502 ,

There will be no additional charge from Azure Databricks End.

If you are saving the data into Azure Storage Account, then you will be billed for the amount of data stored.

You need to pay data transfer out from Amazon S3 to internet.

From Amazon S3 pricing page, here is the data transfer cost.

Hope this helps. Do let us know if you any further queries.

----------------------------------------------------------------------------------------

Do click on "Accept Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2020-07-27T05:24:47.7+00:00

Hello @MH-3502,
Just checking in to see if the above answer helped. If this answers your query, do click “Accept Answer” and Up-Vote for the same. And, if you have any further query do let us know.
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2020-07-29T05:13:50.393+00:00

Hello @MH-3502,
Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
MH-3502 21 Reputation points

2020-07-29T10:18:32.197+00:00
Hi @PRADEEPCHEEKATLA ,
sorry for late respone - unfortunately didn`t receive the notification that there was an answer.

Thank you for the answer - does it mean if i create a dataframe from a S3 bucket containing e.g. 5GB of parquet files

this 5GB will be transferred to Azure environment (in some kind of temp storage)

i wil be billed for this 5GB (every time i read the files again)

?

Thank you!
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2020-07-30T11:43:16.2+00:00

Hello @MH-3502,
You pay for requests made against your S3 buckets and objects. S3 request costs are based on the request type, and are charged on the quantity of requests. There will be no additional charges from Azure. If you are storing the data into Azure Storage Account, then you will be billed for the amount of data stored.
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2020-08-04T10:54:17.24+00:00

Hello @MH-3502 ,
Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
Raj Rao 1 Reputation point

2021-04-01T18:38:46.473+00:00

Hi,

When using Azure Databricks with Azure Storage account, I see that you said that there will be storage costs ("If you are saving the data into Azure Storage Account, then you will be billed for the amount of data stored."), but I was wondering if there any egress costs for that? Would hosting Azure Databricks in a different region than the Storage Account, cause an egress cost?
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2021-04-05T13:04:14.363+00:00

Hello @Raj Rao ,

Every Azure Databricks workspace associated with the managed resource group. This Azure-managed group of resources allows Azure to provide Databricks as a managed service. Initially this managed resource group will contain only a few workspace resources (a virtual network, a security group and a storage account).

If you are utilize the default storage account for storage data it will be charged for the amount of data storage along with ingress/egress charges for accessing data.

Note: Azure Databricks workspace and the managed resource group will be in same regions.

Hope this helps.

Share via

Azure Databricks read from S3 - costs

0 additional answers

Your answer