Hi, I'm a little confused. I'm budgeting a solution that uses Azure Data Factory and Synapse, so I use the Azure Price Calculator. These are my questions: ¿The cluster used in debug session is different from the specified in the integration runtime? I ask this because I see two configuration clusters, that is: I get confused with these two configurations. ¿How is it billed? When using Azure Calculator, I just got this: The first time I said: "Ok, here I can price the spark cluster". but when I consult here , Azure Calculator expresess in terms of vCores and the docs say "nodes" ¿would you help on this, to understand the diferent cluster configuractions (debug, session) and how to locate them in the Azure Calculator? Best regards

Hello Jona , Welcome to the Microsoft Q&A forum. For your first question: Yes, The cluster(IR) used in the debug session is different from the one specified in the dataflow integration runtime (Or) You can use the same IR on the "dataflow debug" session. When you enable "dataflow debug", You will be charged on that particular IR for the duration. Ex: When you turn on "Data flow Debug" on the integration runtime drop-down, you can choose one of the existing IRs you already created. If you choose the debug time to live = 4 hours for a small cluster, you will be charged for the 4 hours on that IR. So your charge on the "dataflow debug" is as follows: small cluster = 8 Vcores total time = 4 hrs 1 debug session = 1 instance total cost = 1 instance * 4 hours Considering your cluster is using basic V-cores, then the cost would be = $8.74 (for the dataflow debug session) My understanding is Basic Vcores are used on the auto IR. (but this is not documented). The billing team can confirm this if you need more details on it. For your question on the document: "Nodes" in the document is nothing but the cluster size( small, medium, large etc) Small node = 4 driver nodes + 4 worker nodes = 8 Vcores Coming to IR on the dataflow: You can also choose the existing IR from the drop-down or create a new IR here. IR is nothing but compute to run the activities. The price varies depending on the IR size- small, medium, large etc. Small nodes have fewer Vcores compared to medium or large nodes. The same above formula applies to the Dataflow IR as well. Ex: If your data flow ran for 4 hours with a small cluster, the price would be the same as above. So, the total price for the dataflow IR = cost of the IR on the "dataflow debug" + cost of the IR on the actual dataflow to run the activities. Here, I am considering a different IRs on the "dataflow bebug" and the actual dataflow. If you use the same cluster, the compute will be shared between the dataflow debug and the dataflow to run the activities. Please note: The minimum vcores for IR is 8 Data Flows Data Flows are visually-designed components that enable data transformations at scale. You pay for the Data Flow cluster execution and debugging time per vCore-hour. The minimum cluster size to run a Data Flow is 8 vCores. Execution and debugging charges are prorated by the minute and rounded up. Reference document: https://azure.microsoft.com/en-us/pricing/details/synapse-analytics/ I hope this helps. Please let me know if you have any further questions. If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.

Spark Cluster configurations and billing on Data Factory/Synapse

Accepted answer

Bhargava-MSFT 31,011 Reputation points Microsoft Employee

2023-09-07T22:48:03.4266667+00:00

Hello Jona,

Welcome to the Microsoft Q&A forum.

For your first question:
Yes, The cluster(IR) used in the debug session is different from the one specified in the dataflow integration runtime

(Or) You can use the same IR on the "dataflow debug" session.

When you enable "dataflow debug", You will be charged on that particular IR for the duration.

Ex:

When you turn on "Data flow Debug" on the integration runtime drop-down, you can choose one of the existing IRs you already created.

If you choose the debug time to live = 4 hours for a small cluster, you will be charged for the 4 hours on that IR.

So your charge on the "dataflow debug" is as follows:

small cluster = 8 Vcores

total time = 4 hrs

1 debug session = 1 instance

total cost = 1 instance * 4 hours

Considering your cluster is using basic V-cores, then the cost would be = $8.74 (for the dataflow debug session)

My understanding is Basic Vcores are used on the auto IR. (but this is not documented). The billing team can confirm this if you need more details on it.

For your question on the document:

"Nodes" in the document is nothing but the cluster size( small, medium, large etc)

Small node = 4 driver nodes + 4 worker nodes = 8 Vcores

Coming to IR on the dataflow:

You can also choose the existing IR from the drop-down or create a new IR here. IR is nothing but compute to run the activities.

The price varies depending on the IR size- small, medium, large etc. Small nodes have fewer Vcores compared to medium or large nodes.

The same above formula applies to the Dataflow IR as well.

Ex: If your data flow ran for 4 hours with a small cluster, the price would be the same as above.

So, the total price for the dataflow IR = cost of the IR on the "dataflow debug" + cost of the IR on the actual dataflow to run the activities.

Here, I am considering a different IRs on the "dataflow bebug" and the actual dataflow.

If you use the same cluster, the compute will be shared between the dataflow debug and the dataflow to run the activities.

Please note: The minimum vcores for IR is 8

Data Flows

Data Flows are visually-designed components that enable data transformations at scale. You pay for the Data Flow cluster execution and debugging time per vCore-hour. The minimum cluster size to run a Data Flow is 8 vCores. Execution and debugging charges are prorated by the minute and rounded up.

Reference document:
https://azure.microsoft.com/en-us/pricing/details/synapse-analytics/

I hope this helps. Please let me know if you have any further questions.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.
Please sign in to rate this answer.

1 person found this answer helpful.
Jona 475 Reputation points

2023-09-08T13:11:05.97+00:00

Hi @Bhargava-MSFT

Considering your cluster is using basic V-cores, then the cost would be = $8.74 (for the dataflow debug session)

¿Where do you configure vCores of your instance? As for now, I've just see # of nodes only

So, the total price for the dataflow IR = cost of the IR on the "dataflow debug" + cost of the IR on the actual dataflow to run the activities. Here, I am considering a different IRs on the "dataflow bebug" and the actual dataflow. If you use the same cluster, the compute will be shared between the dataflow debug and the dataflow to run the activities.

by "shared" ¿what do you mean? Suppose I debug 4 hrs and the run the activity, using the same cluster. That would be 8 hrs of charges ... ¿how that would be billed?

Small node = 4 driver nodes + 4 worker nodes = 8 Vcores

¿Where is this documented?

The first time I said: "Ok, here I can price the spark cluster". but when I consult here, Azure Calculator expresess in terms of vCores and the docs say "nodes"

When you see the calculator expressed as vCores and the docs in nodes, the numbers of nodes doesn't equal or match the vCores offered in the calculator

Best regards

Bhargava-MSFT 31,011 Reputation points Microsoft Employee

2023-09-08T19:19:39.86+00:00

Hello Jona,

¿Where do you configure vCores of your instance? As for now, I've just see # of nodes only

For the above question, the configuration of the vCores on an IR depends on the node size.

Ex: Small node has 8 vcores( 4 driver cores+ 4 worker nodes)

¿what do you mean? Suppose I debug 4 hrs and the run the activity, using the same cluster. That would be 8 hrs of charges ... ¿how that would be billed?

Using the same IR:

Consider using a small node for the "debug session" with TTL = 4 hrs

; the same IR is used on the dataflow for 4 hrs.

You will be charged for 8 Vcores * 4 hrs in this scenario.

The compute(vCores) are shared between the "debug session" and the dataflow activities.

Using different IR:

Consider using a small node(IR1) for the "debug session" with TTL = 4 hrs

, and a different IR(IR2) with the small node is used on dataflow for 4 hrs.

You will be charged for 8 Vcores * 4 hrs (for dataflow debug) + 8 Vcores * 4 hrs (for the dataflow IR)

¿Where is this documented?

Please see the below document. The document talked about the "debug session" cost. My understanding is this is still applicable for synapse.

https://learn.microsoft.com/en-us/azure/data-factory/pricing-examples-mapping-data-flow-debug-workday#azure-data-factory-engineer

When you see the calculator expressed as vCores and the docs in nodes, the numbers of nodes doesn't equal or match the vCores offered in the calculator

You will need to calculate the vCores based on the node size.

For this point: "the numbers of nodes doesn't equal or match the vCores offered in the calculator."

Can you please give me an example where you are seeing the discrepancy?

I hope this helps.

Jona 475 Reputation points

2023-09-09T02:40:50.5733333+00:00

Hi @Bhargava-MSFT

I start to understand a little bit more. With the example you gave me here and reading here and here I know now that dataflow execution and debugging location in the price calculator is this

However, all the docs I red talk about "General Purpose" or "Memory optimized", and these twos flavours (Basic and Standard) are the only ones that appear in the calculator (see the ADF Calculator).

You will need to calculate the vCores based on the node size. For this point: "the numbers of nodes doesn't equal or match the vCores offered in the calculator." Can you please give me an example where you are seeing the discrepancy?

Let me show you. If you see the above instances in the picture, you will see that the only offered vCores configs are 8, 12, 20, 36 y 68. In the other hand, when you read this, you will realize that the total nodes offered won't match the vCores listed prevoiusly.

What's more, I haven't been able to find how much RAM this instances have.

You will be charged for 8 Vcores * 4 hrs (for dataflow debug) + 8 Vcores * 4 hrs (for the dataflow IR)

This will happen if and only if the debug session is alive while running dataflow pipelines, right?

Regards

Bhargava-MSFT 31,011 Reputation points Microsoft Employee

2023-09-11T17:25:21.21+00:00

Hello Jona,

It seems there is an issue with the documentation as well as with the "Pricing calculator"

"General Purpose" and "Memory optimized" are the old compute types on the Dataflow run times.

The new Dataflow run times are - small, medium, large, and custom.

This is documented here.

As you pointed out, the pricing calculator only shows the vCores config- 8, 12, 20, 36, 68

It is supposed to be 16, 32, 48, 80, 144, 272.

Also, there is no documentation about Basic and Standard Vcores referenced in the pricing calculator.

What's more, I haven't been able to find how much RAM this instances have.

For your above question, the RAM on the instances is not exposed to the public documentation. But I can check with my internal team and provide you with more details.

This will happen if and only if the debug session is alive while running dataflow pipelines, right?

You will be charged on the debug cluster until TTL expires. For example, if you turn on the debug cluster for 4 hours, you will be charged for the whole 4 hours(cluster alive for the 4 hours). You will be charged accordingly if you use the same cluster for more than 4 hours.

I will reach out to the ADF product team to get the documentation fixed and provide an update to you.

In case you have any further questions/clarifications, please let me know so that I can consolidate everything and reach out to the PG.

Jona 475 Reputation points

2023-09-11T17:57:10.9533333+00:00

I would be gratefull @Bhargava-MSFT . I'm stuck with budgeting a solution and as you said, it's very important to keep the docs/calculator up to date .

For your above question, the RAM on the instances is not exposed to the public documentation. But I can check with my internal team and provide you with more details.

Thanks very much. Knowing de RAM will help to fit our workloads better. In the price calculator, the Spark Pool for notebooks displays the RAM, for example

I will reach out to the ADF product team to get the documentation fixed and provide an update to you. In case you have any further questions/clarifications, please let me know so that I can consolidate everything and reach out to the PG.

It's just the inconsistency about dataflows clusters between docs and calculator the only one I've found.

¿Any updateor info will be posted here? The RAM of the cluster flavours and the cost of the them is very important to me, so I'll appreciate any info so that I can finish budgeting my solution this week

Best regards.

Bhargava-MSFT 31,011 Reputation points Microsoft Employee

2023-09-12T16:44:41.5433333+00:00

Hello Jona,

I have reached out to the ADF product team with the above questions. I will get back to you as soon as I hear something from them.

Thank you for your patience.

Bhargava-MSFT 31,011 Reputation points Microsoft Employee

2023-09-13T19:43:27.44+00:00

Hello Jona,

Here is an update from the PG team.

Standard and Basic terminology is only used in Synapse. ADF still uses General Purpose and Memory Optimized terminology.

Small, medium, and large are default categories that customers can choose, but customers can still switch to advanced and choose the VM category as General Purpose or Memory Optimized.

Regarding RAM on each instance:
We do not discuss the RAM used by each type as that is an implementation detail, and data flows are intended to abstract the Spark cluster from the user. If customers want to get to that level of detail, PG suggested using Spark Notebooks directly instead of data flows.

Currently, PG is working on the items below.

ADF pricing page will be updated with the correct Vcore details.

Wording on the pricing calculator will be changed to General purpose and Memory optimized instead of the Basic and standard vCores.

I hope this helps.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.

Jona 475 Reputation points

2023-09-13T20:32:13.24+00:00

Hi @Bhargava-MSFT

Standard and Basic terminology is only used in Synapse. ADF still uses General Purpose and Memory Optimized terminology.

So... Does that mean that Basic equals General Purpose an so on? I get confused a little since what I understood from our above conversation is that these cluster types/flavours are old, and the price calculator is not up to date with respect the docs (Small, Medium and so on)

This screenshot is from one of my ADF workspace, and the basic and standard cluster doesn't appear in dataflow cluster configuration.

I'll appreciate if you can enlighten me.

Wording on the pricing calculator will be changed to General purpose and Memory optimized instead of the Basic and standard vCores.

Ohh I see. However, I wonder if you can confirm me if basic and standard are old wordings for the clusters. As I told you, I got confused a little (see first block quote in this comment)

We do not discuss the RAM used by each type as that is an implementation detail, and data flows are intended to abstract the Spark cluster from the user. If customers want to get to that level of detail, PG suggested using Spark Notebooks directly instead of data flows.

I see, so I'll have to through an experimentation phrase to have an idea of the processing powers of the differents vCores flavours

Best regards

Jona

Bhargava-MSFT 31,011 Reputation points Microsoft Employee

2023-09-13T21:11:32.5566667+00:00

Yes, your understanding is correct.

Currently, Basic equals to General purpose and

Standard is equal to Memory Optimized.

If we select small, medium, or large, then the compute type equals - General Purpose(which is standard as per the pricing calculator)

Sorry, my earlier understanding is wrong. General Purpose and Memory optimized are still valid for ADF. They are not old cluster types( I got confused by the small, medium and large clusters earlier)

Regarding your screenshot:

As I explained, Basic and standard Vcore wording is wrong with respect to ADF. They are supposed to be General Purpose and Memory Optimized.

I hope this helps.

Jona 475 Reputation points

2023-09-13T22:19:59.5866667+00:00

thanks @Bhargava-MSFT

Pls, let me know which answer I should mark as correct.

Regards

Jona

Bhargava-MSFT 31,011 Reputation points Microsoft Employee

2023-09-14T00:23:39.09+00:00

Hello Jona,

Thank you. I have converted your answer into a comment. If you could accept my current answer, it would help the community looking for similar issues.

It was a great pleasure working with you on this case.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.